Systematic safety information gaps in AI-generated melanoma therapy information: An exploratory multi-platform evaluation.

Abstract

e13704 Background: Patients and clinicians increasingly use large language models (LLMs) for pharmaceutical information. Prior studies have evaluated AI chatbot accuracy for drug information and oncology applications, finding variable performance with notable hallucinations. However, whether consumer AI platforms adequately communicate critical safety information for oncology therapies has not been systematically characterized. Methods: We conducted an exploratory pilot evaluation of AI-generated responses for 8 FDA-approved melanoma therapies across 3 platforms (ChatGPT/GPT-4-Turbo, Claude/Sonnet, Gemini/2.0-Flash). A total of 6,941 responses were collected and scored against FDA prescribing information using automated LLM-based concordance assessment (Claude Opus 4.5). This pilot used automated scoring without human validation. Results: Our primary finding was systematic safety information gaps across all evaluated drugs. Responses were flagged for missing safety information in 80-85% of cases for established therapies and 99% for the recently-approved TIL therapy Amtagvi. Overall concordance was 83.5% (range: 78.9%-86.3%). We identified 1,137 responses with critical severity flags, including instances where platforms failed to mention serious adverse events. The recently-approved Amtagvi showed the poorest performance, with 98.7% of responses requiring review. We did not assess whether platforms communicated equivalent risk using alternative terminology. Conclusions: This exploratory study identifies systematic safety information gaps as a consistent pattern in AI-generated melanoma therapy information across all drugs and platforms. Findings warrant validation through human expert review to characterize the clinical significance of these omissions.