Mortality in immune TTP remains unpredictable: Shortcomings of prediction models despite machine learning advances

Abstract

Abstract Background Despite advances in the diagnosis and treatment of immune thrombotic thrombocytopenic purpura (iTTP), recent mortality estimates remain around 5–10%. Predicting iTTP mortality risk has been of interest, as it may help individualize treatment or potentially guide the use of more intensive or novel therapies based on disease severity. However, prior attempts at mortality prediction have yielded inconsistent results, and no individual variable has demonstrated independent or reliable predictive ability. In a previous analysis utilizing the USTMA iTTP registry, which included 419 patients from 15 TTP referral centers, we reported that mortality due to immune TTP was unpredictable (Blood. 2021;138[Suppl 1]:1047). Neither the French Thrombotic Microangiopathy (TMA) Reference Score (previously validated in the French TMA Registry), nor a previous logistic regression model attempt (developed by USTMA investigators in 2021) demonstrated sufficient accuracy in predicting iTTP mortality in the USTMA dataset (AUC = 0.63 and 0.69, respectively). We subsequently utilized gradient boosted machine, a machine learning approach capable of handling multiple continuous variables and the complex interactions between them in large datasets, and developed a novel prediction tool: the USTMA Mortality Index (Res Pract Thromb Haemost. 2024; 8(3):102388). It demonstrated good performance in predicting iTTP mortality (AUC = 0.77) in repeated 10-fold cross-validation. However, it has not yet undergone external validation. Objectives To externally validate the USTMA Mortality Index to predict mortality due to acute iTTP in a multicenter cohort. Methods We performed an external validation using an independent cohort of iTTP patients from 5 institutions: University of Utah, Case Western Reserve University, Rochester Regional Health, University of Illinois, and Mayo Clinic. Patients in the validation cohort were not part of the development cohort (the USTMA registry). We included patients with confirmed iTTP and defined acute iTTP mortality as death within 30 days of admission. USTMA Mortality Index variables were collected at initial presentation, which included (in order of highest to lowest relative influence): lactate dehydrogenase, serum creatinine, age, hemoglobin, platelet count, presence of stupor/coma, fever. The model's prediction accuracy was assessed using area under the receiver operating characteristic curve (AUC). We calculated sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) based on the previously defined cut-offs of 4.27% and 9.08% mortality risk in the original model, which categorized patients into standard risk, high risk, and very high risk of predicted mortality. Results The external validation cohort included 103 patients with confirmed iTTP and 11 deaths (10.7% mortality rate). The USTMA Mortality Index demonstrated poor performance with AUC of 0.69 (95% CI, 0.55–0.83) in the validation cohort. The wide confidence interval reflects substantial uncertainty in model performance, ranging from poor to excellent, and is likely due to the small number of deaths. At a risk threshold ≥4.27%:Sensitivity = 64%Specificity = 70%PPV = 20%NPV = 94% At a risk threshold ≥9.08%:Sensitivity = 18%Specificity = 87%PPV = 14%NPV = 90% Conclusion The USTMA Mortality Index demonstrated poor performance in the external validation cohort (AUC = 0.69, 95% CI, 0.55–0.83), contrasting with its previously favorable performance in repeated 10-fold cross-validation. This discrepancy likely reflects model overfitting, a known challenge in AI-based clinical modeling. Our study is limited by the small number of deaths, resulting in a wide confidence interval ranging from poor to excellent performance. Given that neither the French TMA Reference Score, nor our group's prior logistic regression model (2021), nor the USTMA Mortality Index (2024) could reliably predict mortality upon external validation in our cohorts, these findings collectively suggest that mortality in acute iTTP remains difficult to predict at this time, regardless of statistical methodology or current machine learning capabilities. Clinical decision-making, including the use of therapies such as caplacizumab, should not be guided solely by mortality risk predictions. Instead, efforts to reduce mortality and improve outcomes should prioritize individualized clinical assessment over model-based stratification at this time.