Evaluating AI decision support in a rapidly evolving therapeutic landscape: <i>EGFR</i> -mutant metastatic NSCLC.

Abstract

1630 Background: Large language models (LLM) are increasingly used in oncology, yet may deviate from evidence-based care due to outdated knowledge, hallucinations, and inconsistent confidence calibration. EGFR-mutant metastatic Non small Cell Lung Cancer (NSCLC) treatment sequencing is rapidly evolving following 1L OS benefits from MARIPOSA and FLAURA2 and an expanding 2L landscape. In this study, we evaluate concordance, variability, and expert alignment across artificial intelligence (AI) systems. Methods: Twelve expert-curated EGFR-mutant metastatic NSCLC vignettes (6 per 1L and 2L) were evaluated across 4 general-purpose LLMs (ChatGPT, Gemini, Claude, Grok) with deep-research enabled and 2 evidence-based platforms [ASCO AI, OpenEvidence (OpenE)]. Each system simulated percentage-based treatment selections simulating how 100 U.S. med-oncologists might distribute choices across predefined therapies. 5 thoracic oncologists from 3 academic centers established a human reference standard, with unanimous agreement in all but 1 vignette where probability-weighting was applied. Inter-system and expert concordance was assessed using Kendall’s τ, while divergence was quantified using Jensen–Shannon (JS) divergence. Results: Guideline-based platforms showed strong agreement (ASCO AI–OpenE τ=0.73). Among LLMs, ChatGPT aligned the most with ASCO AI (τ=0.72), while Claude showed the highest divergence (JS=0.30); the highest inter-LLM concordance was observed between ChatGPT and Gemini (τ=0.64). Although ASCO AI is built on Gemini, agreement was only moderate (τ=0.55), suggesting added guideline constraints enhance consistency. In 1L, all systems demonstrated moderate-to-strong concordance with experts (τ&gt;0.50), with ChatGPT and ASCO AI the highest (τ=0.80). ASCO AI appropriately emphasized guideline-endorsed options, including Osimertinib monotherapy and combination strategies, without preferential commitment. In 2L, concordance declined substantially with ChatGPT demonstrating inverse correlation with experts (τ=−0.06). Conclusions: AI systems align with expert decision-making in 1L EGFR-mutant metastatic NSCLC but diverge substantially in 2L, where evidence is evolving. Guideline-constrained platforms show greater consistency than general-purpose LLMs; despite ASCO AI using Gemini as its base model, agreement was only moderate, highlighting the value of guideline constraints. These findings highlight the need for rigorous validation and clinical safeguards when integrating AI into oncology decision-making. AI–expert concordance. System 1L - Experts (τ, JS) 2L – Experts (τ, JS) Max Inter-System (τ) ASCO AI 0.80, 0.14 0.25, 0.39 OpenE (0.73) ChatGPT 0.80, 0.15 -0.06, 0.36 ASCO AI (0.72) OpenE 0.67, 0.25 0.27, 0.24 ASCO AI (0.73) Gemini 0.61, 0.21 0.49, 0.19 ChatGPT (0.64) Claude 0.59, 0.32 0.27, 0.22 ChatGPT (0.60) Grok 0.54, 0.26 0.27, 0.25 OpenE (0.64)