Applying a large language model to detect immune-related adverse events in electronic health records: Effect of integrating prescription and laboratory data on accuracy.

Abstract

12133 Background: Accurate identification of immune-related adverse events (irAEs) is essential; however, manual review of electronic health records poses a significant burden on clinicians. Furthermore, conventional text-only approaches may fail to capture clinically important events. In this study, we evaluated whether a Large Language Model (LLM) with a multimodal approach could improve patient-level identification of irAEs. Methods: Among patients who received immune checkpoint inhibitors (ICIs) monotherapy at National Cancer Center Hospital, Tokyo, Japan between October 2014 and July 2023, a subset of patients was randomly sampled for analysis. A zero-shot prompt was designed to identify irAEs. Using a standalone local LLM (gpt-oss-20b), single-modal analysis utilized clinical notes alone, whereas multimodal analysis additionally incorporated systemic steroid prescriptions and laboratory data. Patient-level predictions were aggregated, using the threshold that maximizes the micro-averaged F1-score. The irAEs were clinically adjudicated by board-certified oncologists. Analyses were limited to major irAEs, and performance was evaluated using precision, recall, and the F1-score (α = 0.05). Results: The median number of cases reviewed per day by physicians was 18, whereas the LLM processed 25. During the study period, 296 patients (207 with irAEs and 89 without irAEs) were randomly sampled. Among patients with irAEs, irAE types included rash, pneumonitis, endocrine disorders, colitis, and hepatitis. For the identification of irAE occurrence, the precision of the multimodal approach was significantly higher than that of the single-modal approach (88.7% to 83.7%, p = 0.01). In organ-specific analyses, precision for colitis and rash and recall for endocrine were significantly improved. In addition, the multimodal approach reduced false-positive classifications from 33% to 22% and false-negative classifications from 37% to 35%. IrAE onset timing also differed significantly between the two approaches, with earlier identification by the multimodal LLM (−21.9 vs −17.7 days, p = 0.04); both approaches detected onset earlier than clinical adjudication. Conclusions: A multimodal LLM framework improves irAE detection accuracy and may reduce clinician workload. IrAEs with significant differences between single-modal and multimodal analyses. Precision Recall F1-score % 95%CI P % 95%CI P % Any irAE Multi 88.7 84.2-93.1 0.01 83.1 78.0-88.2 0.62 85.8 Single 83.7 78.7-88.8 82.1 76.9-87.3 82.9 Rash Multi 64.5 53.7-75.2 0.03 60.5 49.8-71.1 0.10 62.4 Single 58.9 48.7-69.1 65.4 55.1-75.8 62.0 Colitis Multi 80.4 69.5-91.3 <0.01 77.4 66.1-88.6 0.32 78.8 Single 61.8 50.2-73.3 79.2 68.3-90.2 69.4 Endocrine Multi 71.2 59.6-82.7 0.37 75.0 63.7-86.3 0.03 73.0 Single 76.6 64.5-88.7 64.3 51.7-76.8 69.9