Using a medically trained LLM-based end-to-end system in automating patient eligibility screening across a health system for A phase 3 study evaluating the safety of givinostat in patients with polycythemia vera.

Abstract

Abstract Background: The process of assessing and identifying eligible participants for clinical trials is lengthy, and resource-intensive, placing a burden on clinical research systems already strained by staffing shortages. The challenge is even more profound in the setting of rare diseases like polycythemia vera (PV) a chronic blood cancer with a prevalence of 44-57 per 100,000 people in the US. Traditionally, the chart review entails screening by human clinicians and research nurses requiring >30 minutes of review per EMR depending on the situationally specific nature of the review including (1) thoroughness of the review (reading, skimming, or targeted), (2) number of questions or criteria for which a conclusion must be deduced, (3) extent of history reviewed (3, 6, 12 months, etc.). Trial recruitment is often limited to patients seen in routine clinical care, making the recruitment of patients with rare diseases, particularly challenging. LLMs embedded within EHRs can transform the efficiency and scope of trial recruitment. We evaluated Synapsis AI, a medically trained, large language model-based (LLM-based) end-to-end system, focusing on its accuracy and efficiency in identifying eligible patients for an active PV clinical trial, being conducted at Cleveland Clinic. Methods: The trial is a randomized, phase 3 study comparing givinostat versus hydroxyurea (NCT06093672). Synapsis AI pre-screened patients for the clinical trial, a process typically carried out manually by nurse coordinators. The AI system leverages structured data and LLM-based interpretation of free-text clinical notes. After filtering oncology-related ICD_10 codes from the prior 3 years, patients with a diagnosis of PV were isolated by the LLM component, which then assessed each patient's eligibility against the trial's 7 inclusion and 20 exclusion criteria using both structured and unstructured data elements. We assessed the ability of Synapsis AI accuracy and positive predictive value (PPV) in identifying patients eligible for the GIV-IN PV study accurately and compared the volumes of patients identified by Synapsis AI versus the traditional screening process. Results: Out of 4.7 million EMRs active at the Cleveland Clinic's database, 28,200 patients with an oncology diagnosis within the past 3 years were identified. Of these, 904 were found to have a diagnosis of PV. Synapsis AI completed full eligibility assessments on these 904 patients within one week, against the trial's criteria and identified 22 eligible patients. Subsequent reviewed by the Cleveland Clinic's research personnel verified eligibility (100% PPV), confirming 100% accuracy of the system's assessments against the protocol's criteria. In contrast, the traditional manual approach, involving clinicians reviewing their schedule and nurses pre-screening patients, pre-screened 9, enrolled 4, and treated 3 patients over a 12-month period, with and average enrollment of approximately 1 patient every 4 months. Synapsis AI's approach therefore demonstrated a 7-fold increase in eligible patient identification within a fraction of the time, substantially reducing workload and accelerating timelines. Conclusion: This comparison demonstrates the potential of a medically trained LLM-based system for accelerating and expanding patient identification and recruitment for clinical trials, significantly reducing the time and effort required. The stark contrast in the number of patients identified highlights the significant efficiency and time-saving advantage of Synapsis AI's automated pre-screening over conventional screening methods. By streamlining the recruitment process, tools like Synapsis AI can enhance trial efficiency and support faster development of life-saving therapies even in the setting of rare diseases.