Predicting immune-related adverse events (irAEs) with large language model (LLM) embeddings.

Abstract

1611 Background: irAEs are a common side effect of immune checkpoint inhibitor (ICI) therapy. Predicting irAEs from clinical records relies on traditional machine learning methods, which require time-consuming and extensive feature engineering such as removing features with many missing values or imputing values. Using LLM vector embeddings to train classifier models may require less feature engineering and leverage LLM understanding of semantic information. We assessed whether this LLM method could predict development of irAEs 12 weeks after initiating an ICI. Methods: We used patient (pt) data from the Immuno-Oncology Registry, a database of cancer pts from 10 DC-Baltimore-based MedStar Health network hospitals and Hackensack Meridian Health system in New Jersey. The data were divided into training (80%) and testing (20%) datasets. The training dataset was oversampled to balance the irAE class distribution. Tabular data was serialized into natural language, then passed into a LLM with original text, 0-shot, 1-shot, or few-shot prompting. Due to LLM context length limits, pt information was separated into 9 components, and each component embedding was max-pooled before concatenation. The embeddings were used to train 4 classifier models with hyperparameter tuning: XGBoost, support vector machine (SVM), logistic regression (LR), and multilayer perceptron (MLP). Five different sentence transformer or text embedding LLMs were tested. Model classification performance was evaluated by Area Under the Receiver Operating Characteristic curve (AUROC). Results: Of the 1459 pts, 585 (40%) experienced irAE(s). The most common cancers were lung cancer (34.5%) and melanoma (27.7%). ICIs included anti-CTLA-4 (11.0%), anti-PD-1 (58.9%), anti-PD-L1 (4.6%), anti-PD-1 + anti-CTLA-4 (14.0%), anti-PD-1 + chemotherapy (3.1%), anti-PD-1 + tyrosine kinase inhibitor (0.07%), and others (8.4%). The most common irAEs were skin rash (14.5%) and colitis (8.9%). Median AUROC was highest for XGBoost (0.654), followed by LR (0.641), MLP (0.627), and finally SVM (0.596). Median AUROC was highest for 0-shot (0.641), followed by original (0.636), 1-shot (0.619), and few-shot (0.516). Of the LLMs tested, intfloat/e5-base combined with XGBoost and the original prompt demonstrated the strongest performance (AUROC = 0.712, see Table), although for 0-shot and 1-shot LR was a stronger classifier. Conclusions: Using LLM embeddings to train classifier models is feasible for making predictions with tabular medical data, works best with XGBoost, and does not need prompt engineering. This method requires less feature engineering and data processing compared to standard approaches and leverages the extensive pre-training and semantic understanding of LLMs to achieve predictive power. e5-base original 0-shot 1-shot few-shot XGBoost 0.71 0.62 0.67 0.62 SVM 0.56 0.61 0.67 0.65 LR 0.64 0.66 0.69 0.64 MLP 0.63 0.65 0.65 0.59