Clinical large language model to predict loss to follow up for oncology patients discharged to skilled nursing facilities.

Abstract

e13597 Background: From SEER data, there is evidence that oncology patients who are discharged to a skilled nursing facility are less likely to receive chemotherapy and radiation. Those that are unlikely to follow up are a vulnerable population that should be identified to discuss how to best optimize post hospitalization care. Methods: Through University of Florida (UF)’s Integrated Data Repository of EHR data, we identified 591 hospitalization events for UF established oncology patients that lead to a discharge to a skilled nursing facility from UF/Shands Hospital between 2012 – 2023. The outcome of interest was loss to follow up, defined as no clinic follow up within 6 months after discharge. 222 hospitalization events lead to loss to follow up and 369 did not. We trained XGBoost, random forest and logistic regression models on structured EHR data such as ICD codes, patient demographics including insurance payer, Charlson Comorbidity Index, clinical features such as discharge vitals with a data split of 6:4 for training and testing to predict loss to follow up. We compared this against GatorTron, a clinical large language model, which was trained separately on hospital discharge summaries and concatenated notes from physical therapists, occupational therapists, case managers and social workers with a data split of 6:2:2 for training, validation and testing. Results: As noted in the table, XGBoost performed better than logistic regression or random forest with a F1 score of 0.298. GatorTron_345m using physician notes from hospital discharge summaries, however yielded a poorer performance with a F1 score of 0.229. With additional notes from physical therapists and occupational therapists, case managers and social workers, GatorTron_345m achieved the best performance amongst all the models with a F1 score of 0.377 using a learning rate of 1e-5 for 30 epochs. Conclusions: To the best of our knowledge, there is no study leveraging machine learning, especially large language models to predict loss to follow up in oncology clinics. This is the first study to initiate predictive modelling with respect to this important clinical question. This study also highlights the value of multidisciplinary notes in predicting post-hospitalization outcomes rather than physician notes only. Loss to follow up for oncology patients is a multifactorial issue and despite the collection of a broad selection of patient and encounter features, as well as the employment of advanced machine learning tools in this study, elucidation of better determinants is needed. Model performance results. Input Model F1 Precision (PPV) Recall (sensit) Specificity Accuracy ICD + demographics XGBoost 0.298 0.362 0.254 0.729 0.550 Random Forest 0.137 0.3 0.089 0.873 0.578 LogisticRegression 0.264 0.3 0.236 0.669 0.506 Discharge Summaries GatorTron_345m 0.229 0.444 0.154 0.882 0.606 Multidisciplinary Notes 0.377 0.482 0.310 0.825 0.648