Advancing AI in oncology: A performance comparison of ChatGPT-4o and ChatGPT-o1 in neuroendocrine tumor clinical decision making.

Abstract

e13730 Background: Artificial intelligence (AI) is increasingly integrated into clinical decision-making, offering potential support in complex oncologic management. While ChatGPT (developed by OpenAI, San Francisco, CA) has demonstrated utility in various medical applications, its ability to accurately navigate guideline-based recommendations for neuroendocrine tumors (NETs) remains unexplored. Objective: This study aimed to assess the comparative performance of GPT-4o and GPT-o1 in providing guideline-based recommendations for NET management, utilizing the National Comprehensive Cancer Network (NCCN) guidelines as a reference standard. Methods: A total of 43 clinical questions were systematically derived from the NCCN guidelines, covering key decision-making domains: treatment, surveillance, and diagnostics. Both GPT-4o and GPT-o1 were queried, with responses independently evaluated by two physicians using a Likert scale (1-5), where: 5 = Fully correct, 4 = Mostly correct (minor omissions), 3 = Partially correct (lacking completion), 2 = Partially incorrect, 1 = Completely incorrect. Results: To explore the interpretative difference of ChatGPT-4o and ChatGPT-o1 on neuroendocrine tumor management, a Mann-Whitney U test was conducted. The analysis was statistically borderline (p = .050). The Mean Rank of ChatGPT-o1 was greater than that of ChatGPT-4o, thus GPT-o1 can be considered superior to GPT-4o. (Mann-Whitney U-test; Mean Rank = 47.16 vs 39.84) When stratified by question category, no statistically significant differences were observed in diagnostics, treatment, or surveillance. Conclusions: ChatGPT-o1 exhibited marginal improvements in the decision making for NET management while maintaining comparable performance to ChatGPT-4o in other domains. These findings suggest that iterative advancements in AI models may enhance their ability to support evidence-based clinical decision-making. Further studies are warranted to validate these findings in real-world oncologic practice.