Abstract 4364716: Artificial Intelligence Analysis of Free-Text Discharge Summaries Facilitates Automated Risk Stratification for Cardiac Surgery Readmissions

Abstract

Background: Cardiac surgery patients at high risk of readmission can be identified using risk calculators that rely on structured features requiring time-consuming manual chart review. We aimed to develop a cardiac surgery-specific artificial intelligence (AI) model to predict 30-day readmission using a large language model (LLM) to analyze electronic clinical notes. Secondary aims were to evaluate temporal trends in model performance, given changes in practice and the increasing use of short-stay units, and to correlate risk-predicted scores with readmission length of stay (LOS). Methods: We fine-tuned a general medical LLM developed at our institution using discharge summaries of 8,275 adult cardiac surgery patients (2014-2024). We implemented strict temporal validation by reserving post-June 2021 cases (n=3,031) as a pseudo-prospective test cohort. To evaluate temporal generalization, we assessed performance degradation over time (2022-2024) and investigated recalibration approaches. The model's potential as a general biomarker of post-cardiac surgery risk was evaluated by testing discrimination between observation stays (<48 hours) versus inpatient readmissions, and analyzing relationships between predicted risk scores and readmission LOS. Results: Our model achieved a temporal area under the receiver operating characteristic curve (AUC) of 0.71, with sensitivity 78.5% and specificity 50.1% (Figure 1). Performance showed temporal degradation (AUC 0.78 to 0.67) from 2022-2024, mitigated via recalibration. Risk scores predicted readmission LOS (p<0.001) and discriminated between observation and inpatient stays (AUC 0.69). We observed a bimodal risk distribution, suggesting distinct readmission risk phenotypes. Figure 2 shows that patients in the highest risk quintile experienced a mean LOS 1.9 times longer than the lowest quintile (7.1 vs. 3.7 days, p<0.001). Predicted risk scores increased annually, suggesting documentation drift may impact model performance (Figure 3). Conclusions: This study demonstrates that a cardiac-specific AI model, using analysis of unstructured clinical notes, predicted 30-day readmissions and captured key features of patient risk profiles without task-specific fine-tuning. Our analyses support the use of recalibration methods to mitigate data drift in a realistic deployment setting. By developing an LLM using postoperative cardiac surgery records, automated screening for high-risk readmissions becomes possible before discharge.