Artificial intelligence for immunotherapy response assessment in lung cancer using PET-CT reports.

Abstract

8577 Background: Accurate and timely assessment of immunotherapy response is vital for optimizing lung cancer management. This study evaluates the efficacy of a large language model (LLM), Gemini 1.5 Pro, in automating response assessment using positron emission tomography/computed tomography (PET/CT) reports based on the European Organization for Research and Treatment of Cancer (EORTC) criteria. Methods: Google Gemini 1.5 Pro was selected due to its large context window capacity and its free availability via the web interface.The model was utilized with explicit instructions on applying EORTC criteria and fine-tuned using few-shot prompting. Pre- and post-immunotherapy PET-CT reports in text format from 33 lung cancer patients, anonymized in compliance with HIPAA regulations, were independently classified by the LLM and an experienced nuclear medicine specialist. Performance metrics, including precision, recall, F1-score, and support, were calculated for each response category. Inter-rater agreement was assessed using Cohen's Kappa. Results: The nuclear medicine specialist classified 5, 21, 6, and 1 cases as complete metabolic response (CMR), progressive metabolic disease (PMD), partial metabolic response (PMR), and stable metabolic disease (SMD), respectively, while Gemini 1.5 Pro classified 5, 20, 7, and 1 cases accordingly. The LLM achieved an overall accuracy of 97% and demonstrated excellent agreement with the expert (overall Cohen's Kappa: 0.945). F1-scores were 1.00 for CMR and SMD, 0.98 for PMD, and 0.92 for PMR, with per-label Kappa scores ranging from 0.904 (PMR) to 1.00 (CMR and SMD) (Table 1). Conclusions: Gemini 1.5 Pro exhibits strong potential for automating accurate immunotherapy response assessment in lung cancer using PET-CT reports. Its high concordance with expert evaluations underscores its utility in streamlining clinical workflows and improving efficiency. Validation with larger, more diverse datasets is warranted to support its clinical implementation. Performance metrics of Gemini 1.5 Pro for immunotherapy response assessment. Response Precision Recall F1-score Support Cohen’s Kappa CMR 1.00 1.00 1.00 5 1.000 PMD 1.00 0.95 0.98 21 0.936 PMR 0.86 1.00 0.92 6 0.904 SMD 1.00 1.00 1.00 1 1.000 CMR, Complete Metabolic Response; PMD, Progressive Metabolic Disease; PMR, Partial Metabolic Response; SMD, Stable Metabolic Disease.