Analysis of a large language model-based system versus manual review in clinical data abstraction and deduction from real-world medical records of patients with melanoma for clinical trial eligibility assessment.

Abstract

1571 Background: Manual chart review (MCR) is the gold standard for assessment of information from electronic medical records (EMRs) for clinical trial eligibility. However, this method is labor-intensive, prone to error, and limited in scalability with high volumes of unstructured EMR data. Large language models (LLMs), have shown promise in natural language understanding, and automating chart review and abstraction would significantly improve efficiency and accuracy in data review for clinical research. In this evaluation project, we compared the performance of Synapsis LLM, a medically-specialized LLM, with medical professionals at answering questions tied to eligibility criteria of relevant clinical trials, by reading clinical notes of patients with melanoma. Methods: We conducted a comparative analysis using records of randomly selected patients with melanoma from the Cleveland Clinic. Two parallel processes were assessed: (1) MCR conducted by a melanoma and an oncology specialized research nurse (2) Automated chart review using the Synapsis LLM. Both processes ran on two cohorts: Cohort (A) consisting of 25 EMRs that were posed 23 eligibility questions each, and Cohort (B) consisting of 25 different EMRs, posed 22 eligibility questions each. In total, there were 1,125 questions answered by each of the research nurses as well as Synapsis AI. The questions addressed focused on melanoma-specific clinical characteristics, including but not limited to treatment approaches, related surgical procedures, imaging findings, and genetic testing. Performance metrics included accuracy of answers to the questions, and time required to complete the abstraction process. Discrepancies between the responses of the two research nurses and the LLM were analyzed in comparison to the established ground truth, which was determined through a consensus review by physicians to ensure the validity and reliability of the results. Results: Synapsis LLM performed the task with 95.73% accuracy in 2.5 minutes while the melanoma specialized nurse responded with 95.11% accuracy in 427 minutes. The oncology specialized research nurse’s accuracy was 88.09%, and the tasks was completed in 540 min. The comparison demonstrated significant time savings and medical-grade accuracy for the application of this LLM-based technology compared to manual methods. Conclusions: This is the first project that compares an LLM-based system vs research nurses in deducing clinical characteristics from patients’ EMRs for clinical trial eligibility. Synapsis LLM accurately completed the abstraction process, outperforming in accuracy and time the clinical personnel. This study highlights the potential of LLMs like Synapsis AI in scalable clinical research applications that currently rely solely on MCR.