Understanding patient engagement with an oncology chatbot: Thematic and sentiment analysis of medical adherence.

Abstract

e13705 Background: Oral oncology treatment is an effective option, but faces challenges of suboptimal adherence. Penny, an oncology support chatbot, was designed to enhance patients’ engagement and symptom management. While the parent pilot randomized controlled trial showed non-significant effects of the intervention on medical adherence, traditional qualitative analyses showed favorable user experiences. To better explain these discrepancies, we leverage large language models (LLMs) to extract latent emotional, thematic, and sentiment signals from patient interviews to understand how patients emotionally and behaviorally engaged with Penny during routine oral oncology treatment support over a 12-week period. Methods: Of the 38 patients randomized to the Penny arm, 19 completed semi-structured interviews. Latent Dirichlet Allocation (LDA) was applied to 4,835 cleaned words derived from a combined term frequency-inverse document frequency (TF-IDF) and bigram dataset of interview transcripts. Sentiment analysis using a BERT-based model was conducted on 1,089 cleaned interview sentences. Generalized linear mixed-effects logistic regression was used to examine fixed effects of sentiment polarity, medication user type, and adherence, with participant-level random effects. Results: Thematic analysis identified two medical adherence-related aspects: 1) Emotional support, such as anxiety reduction and confidence improvement in managing treatment; and 2) Routine practical assistance, such as reminders, symptom monitoring, and appointment coordination. A crude descriptive analysis showed that adherent patients had a significantly higher positive sentiment (29.1% vs. 20.4%, p = 0.012). Stratified descriptive analysis showed that prevalent users, defined as those already using the medication at enrollment, had higher positive sentiment towards Penny than newly prescribed users in both the adherence and non-adherence subgroups. This difference was statistically significant only among the adherent group (32.7% vs. 26.3%, p =0.033). The mixed-effects model showed no significant association between sentiment and medical adherence at the fixed-effect level, and the random effects had a low adjusted ICC (0.029). Conclusions: The study suggests that patient engagement with an oncology chatbot is associated with both emotional and practical support needs and treatment history. Higher positive sentiment among prevalent users suggests that prior medication experience may play an important role as an associated factor in patients' emotional response to digital support tools, with potential implications for medical adherence. The high variation within each patient indicates that adherence engagement is a dynamic, context-dependent process, which requires an implementation-focused, patient-centered strategy to better support oral oncology treatment. Clinical trial information: NCT04347161 .