Validation of an AI-based clinical decision support system for guideline-concordant prostate cancer treatment at a Brazilian cancer center.

Abstract

1605 Background: Clinical decision support systems (CDSS) based on large language models may improve adherence to evidence-based guidelines in prostate cancer. We developed a Gemini-based CDSS for localized and locally advanced prostate cancer treatment at Instituto do Câncer do Ceará (ICC), Fortaleza, Brazil, and conducted a validation study comparing its recommendations with National Comprehensive Cancer Network (NCCN) guidelines and real-world physician decisions. Methods: We retrospectively included 198 consecutive patients prostate cancer treated at ICC. For each case, the CDSS generated a recommended primary treatment. Independent clinician reviewers assigned reference NCCN-concordant treatment categories. We calculated per-treatment precision, recall, F1-score, and one-vs-rest AUC comparing CDSS and physicians, with particular focus on active surveillance (AS) and radical prostatectomy (RP). Results: Overall NCCN concordance was 87.9% for the CDSS and 82.3% for physicians. Treatment-level analysis showed that the CDSS more often and more accurately identified candidates for active surveillance. For AS, the CDSS achieved precision 1.00, recall 1.00, F1-score 1.00, and AUC 1.00, while physicians had precision 1.00 but substantially lower recall (0.24), F1-score 0.39, and AUC 0.62. In contrast, physicians more frequently recommended RP beyond NCCN indications: for prostatectomy, the CDSS showed precision 1.00, recall 1.00, F1-score 1.00, and AUC 1.00, whereas physicians had lower precision (0.40) with recall 1.00, F1-score 0.58, and AUC 0.90. Conclusions: In this single-center validation study, a Gemini-based CDSS for prostate cancer treatment achieved higher overall NCCN concordance than treating physicians and improved alignment with guidelines for the appropriate use of AS. These findings suggest that integration of large language model–based CDSS into clinical workflows at cancer centers in low- and middle-income settings may reduce overtreatment and support more value-based prostate cancer care. Although we did not include data on social impossibility of AS, the finding leads to reflections on the possibility of greater use of AS, even in a region with limited resources. Performance of a Gemini-based clinical decision support system vs physicians for active surveillance and radical prostatectomy (NCCN-concordant treatment metrics). Treatment modality Metric CDSS (Gemini-based) Physicians Active surveillance (VA) Precision 1.00 1.00 Active surveillance (VA) Recall 1.00 0.24 Active surveillance (VA) F1-score 1.00 0.39 Active surveillance (VA) AUC 1.00 0.62 Radical prostatectomy (P) Precision 1.00 0.40 Radical prostatectomy (P) Recall 1.00 1.00 Radical prostatectomy (P) F1-score 1.00 0.58 Radical prostatectomy (P) AUC 1.00 0.90