Using large language models to assess adherence to ASCO tele-health in oncology standards in post-Covid clinical documentation.

Abstract

e23238 Background: In 2021, the American Society of Clinical Oncology (ASCO) published telehealth standards and practice recommendations outlining appropriate use, workflow, and required documentation for telehealth encounters in oncology. These standards include explicit documentation elements intended to ensure quality, equity, and continuity of care. In the post-COVID era, telehealth remains widely used; however, adherence to ASCO telehealth documentation standards has not been systematically evaluated. Manual chart review of unstructured electronic health record (EHR) notes is time-intensive and limits scalability. Large language models (LLMs) may enable efficient extraction of telehealth adherence measures from free-text clinical documentation. Methods: We conducted a retrospective EHR study of oncology telehealth encounters occurring between January 1, 2024 and October 1, 2025. Eighty patients with advanced solid or hematologic malignancies were randomly selected, each with at least one synchronous telehealth encounter (audio-only or audio-video). Fourteen documentation and appropriateness domains were derived from ASCO 2021 telehealth standards, including required visit documentation elements and visit-selection guidance. Two physician reviewers independently performed manual chart review with adjudication. A HIPAA-secure LLM was prompted to identify each domain and required generation of supporting source text. Performance was evaluated using sensitivity, specificity, accuracy, and F1 score. Results: Across adherence domains, LLM sensitivity ranged from 0.46-0.94, specificity from 0.34-0.97, and accuracy from 0.56-0.95. Highest performance was observed for documentation of visit modality and visit purpose, while lower performance was noted for patient location, provider location, and visit completion. Audio-video encounters demonstrated higher documentation completeness than audio-only encounters across most domains. The overall hallucination index was low (4.2%). Median review time decreased from 9.1 minutes per encounter with manual chart review to 0.8 minutes using LLM-assisted extraction. Conclusions: LLMs can identify adherence to ASCO telehealth documentation standards from unstructured oncology EHR notes in the post-COVID era. This approach may support scalable telehealth quality monitoring, identify systematic documentation gaps, and inform targeted workflow improvements in oncology practice.