Generative artificial intelligence (AI) for predictive analysis: Can AI estimate the likelihood of sentinel lymph node metastases in early-stage breast cancer?

Abstract

e23133 Background: Generative AI model o1 has potential to compile relevant information from pathology and radiology reports, facilitating medical record summarization for clinical application including efficient use of nomograms. Nomograms predicting the likelihood of sentinel lymph node (SLN) metastasis in early-stage breast cancer (EBC) can be beneficial when deciding omission of axillary surgical staging. We explored the ability of AI to summarize medical records accurately in order to reproduce nomogram estimates for SLN metastases. Methods: Patient age and de-identified radiology and pathology reports from 20 EBC patients were provided to o1-preview model. We also supplied o1 with URLs from MD Anderson Cancer Center (MDACC) and Memorial Sloan Kettering Cancer Center (MSKCC) nomogram calculator websites on predicting SLN metastasis. The o1 was tested in 3 different sessions using the following prompt: Learn this nomogram. Show step by step the variables used for calculation, and show the predicted percent estimates for the likelihood of SLN metastasis using the MDACC and MSKCC nomograms. Next, using the same prompt but percent estimates were corrected after each case (serial corrections, SC). Lastly, without a nomogram: estimate the percent likelihood of SLN metastasis in a patient with EBC based on radiology and pathology reports. Outputs were compared to those obtained through manual use of the nomogram website, yielding 120 comparisons. Results: The 7 MDACC and 9 MSKCC nomogram variables were accurately identified in 12 (60%) cases without SC (94% MDACC and 95% MSKCC variables) and in 16 (85%) cases with SC (96% MDACC and 97% MSKCC variables). Without using a nomogram, o1 used 7-9 variables, which were correctly identified 96% of the time. Most common discordance was in tumor size, such as not using the largest size for tumor estimate or using the size from the pathology report. Only once o1 hallucinated the tumor size. Overall, o1 accurately estimated SLN positivity in 3.3% (n = 4) of cases; 2 without nomogram use and 2 using MDACC nomograms with SC. When estimating without a nomogram, o1 tended to underestimate compared to manual calculation of MDACC and MSKCC nomograms, in 60% and 80% of cases respectively. When using the nomogram, o1 also underestimated more without SC (MDACC 60% without SC vs MDACC 50% with SC; MSKCC 90% without SC vs 40% with SC). While not statistically significant, the average deviation in SLN positivity predictions were highest without nomogram use (14.4% without vs 10.9% with nomogram use). Conclusions: The o1 model shows potential to provide accurate clinical summary of breast radiology and pathology reports. The nuance of discerning tumor size impacted the accuracy of SLN metastasis predictions. Further refinement of generative AI tools are needed before they can be reliably integrated into clinical workflows.