Reliability and readability of five AI chatbots for concussion health advice across retrieval augmented and pretrained models

Abstract

Abstract Generative AI is rapidly entering patient education workflows, yet its safety profile for concussion management remains undefined. Utilizing the CHART framework, this cross-sectional audit assessed five platforms, specifically isolating retrieval-augmented generation (RAG) architectures against standard pre-trained Large Language Models (LLMs). We extracted 11 high-volume patient queries from Google Trends and administered them via a zero-shot protocol. Two blinded neurosurgeons then scored the outputs against the 2023 Amsterdam Consensus Statement using four validated instruments: DISCERN and EQIP to evaluate treatment and information quality, GQS for global content quality, and JAMA benchmarks for transparency. Reliability metrics diverged significantly across models (DISCERN and EQIP, p  < 0.001). Perplexity Pro secured the highest DISCERN (47.36 ± 4.84) and EQIP (65.00 ± 5.48) values, statistically surpassing foundational models like ChatGPT and Gemini ( p  < 0.01) — a performance gap likely driven by its RAG design. In contrast, GQS scores did not differ significantly across models ( p  = 0.373), and JAMA-based transparency remained uniformly low ( p  < 0.001). Readability was assessed using six standard indices (FRES, FKGL, GFI, CLI, ARI, and SMOG), revealing that all models exceeded the 6th-grade reading level; most surpassed 10th-grade, with Perplexity Pro lowest at FKGL = 7.46. Although retrieval-augmented systems improve clinical accuracy, current iterations fail to provide transparent or readable advice. Clinical integration therefore requires rigorous human-in-the-loop verification and a shift toward plain-language algorithm optimization.