Impact of large language models on trial-to-patient matching efficiency.

Abstract

e13600 Background: Timely and accurate clinical trial matching is critical to providing additional treatment options for patients, especially when standard-of-care treatments are limited. However, trial matching is time-consuming, leading to missed patient opportunities and delays in trial enrollment. The Tempus AI TIME Program utilizes a technology-enabled patient matching platform to improve screening efficiency and accelerate trial enrollment. TIME uses structured data followed by manual nurse review to assess patient eligibility. To enhance efficiency, we explored the incorporation of a retrieval augmented generation (RAG) question answering system based on large language models (LLMs) to evaluate additional unstructured data and prioritize patients for review. Methods: Using a phase 2 trial enrolling patients with advanced breast cancer, we identified potentially eligible patients based on high-level criteria including tumor type, stage, and biomarkers. We developed a set of LLM queries to assess additional eligibility criteria, such as the presence of comorbidities, second primary malignancy, or hospice care. For each query and patient, the RAG system retrieved 16 text snippets (approximately 1000 characters each) based on textual similarity to the prompt. These snippets and the query were input into GPT-4o to generate contextually relevant responses. A "match score" was then calculated for each patient (0-1, 1 = all queries satisfied). Nurse review was prioritized based on this score and each patient was assigned a trial status of "match" or "not a match". The match rate and time for patient review were evaluated. Results: Over two months, match scores were generated for 1,015 patients. In the first month, RAG system output and nurse feedback were used for prompt tuning, resulting in a final set of 13 queries. In the second month, these queries were used to optimize the screening of 508 patients. Patient review was prioritized by match score, and the match rate and number of patients matched over time were compared to a hypothetical random order scenario (Table 1). In both scenarios, the same 508 patients are reviewed, starting with 0 matched patients and ending with 66 matches. Patients with a match score of 1 had a 67% match rate compared to the average match rate of 13%. In addition, the optimized order resulted in more patient matches earlier in the process. Conclusions: The TIME program supports oncology practices by accelerating high-quality patient matching for clinical trials. Through a balanced process incorporating both LLMs and nurse verification, TIME demonstrated high efficiency gains for initial patient screening. Matching efficiency. Cumulative Patients Reviewed 21 160 351 473 508 Cumulative Review Time [hours] 3.5 27 59 79 85 Optimized Order  Match Score (s) s = 1 s ≥ 0.9 s ≥ 0.8 s ≥ 0.7 s > 0  Patients Matched 14 40 57 64 66  Match Rate 67% 25% 16% 14% 13% Random Order  Patients Matched 3 21 46 61 66  Match Rate 13% 13% 13% 13% 13%