Effect of incorporating symptom burden with mortality as a composite outcome on accuracy and bias in palliative care identification algorithms in oncology.

Abstract

12016 Background: Machine learning (ML) algorithms are increasingly used to identify patients for early palliative care (PC) or advance care planning (ACP). Most PC/ACP algorithms are trained using only mortality as an outcome. However, increasing availability of structured patient-reported outcomes (PROs) in electronic health record (EHR) databases can facilitate more comprehensive identification of palliative care need by training algorithms on composite outcomes of mortality and symptom burden. Methods: Our cohort consisted of patients with cancer seen at one of 18 practices in 2019 within a large academic cancer center. We leveraged structured EHR data, consisting of 153 demographic, laboratory, and comorbidity features and 12 symptom scores derived from CTCAE-PRO that were routinely reported at medical oncology encounters (72% response rate). Our Base Model was a random forest model predicting 180-day mortality from the date of an initial medical oncology encounter (index encounter) that was used in practice to prompt earlier ACP conversations. We retrained models using a Composite Label of mortality and/or severe symptoms (≥3 out of 5 in at least 1 symptom) within 180-days of an index encounter. We report performance for Base vs. Retrained models in 1,000 bootstrapped samples predicting the Composite Label using area under the precision-recall curve (AUPRC) and true positive rate (TPR) for All Patients, Black Patients, and White Patients. We hypothesized that Retrained models would improve performance and reduce Black-White disparities. Results: Our cohort consisted of 4908 patients (median age 64.1 years [IQR 17.5], 53.0% female, 59.6% solid tumor malignancies). Retrained Models improved TPR over Base Models for All (0.56 [95% CI 0.52-0.59] vs. 0.18 [95% CI 0.15-0.21]), Black (0.60 [95% CI 0.52-0.67] vs. 0.20 [95% CI 0.14-0.26]), and White (0.55 [95% CI 0.50-0.59] vs. 0.17 [95% CI 0.14-0.20]) patients, with similar AUPRC. TPR improvement was marginally greater for Black vs. White patients (0.02 [95% CI -0.01-0.05]). Conclusions: In this cohort study, retraining a PC identification algorithm on a Composite outcome of symptom burden + mortality significantly enhanced identification of PC need, with disproportionate improvements for Black patients. Incorporating PROs into PC identification model outcome labels should be strongly encouraged. Comparison of base vs. retrained model performance. All Patients a Black Patients a White Patients a Other b Model Base Retrained Base Retrained Base Retrained Difference in Black vs. White Improvement AUPRC 0.71 (0.68, 0.74) 0.71 (0.68, 0.74) 0.80 (0.73, 0.85) 0.79 (0.72, 0.85) 0.68 (0.65, 0.72) 0.68 (0.65, 0.72) 0 (-0.01, 0.08) TPR 0.18 (0.15, 0.21) 0.56 (0.52, 0.59) 0.20 (0.14, 0.26) 0.60 (0.52, 0.67) 0.17 (0.14, 0.20) 0.55 (0.50, 0.59) 0.02 (-0.01, 0.05) a Mean (95% CI) b Mean difference (95% CI).