Implementing a retrieval-augmented generation-based large language model to guide oncologists in searching for FDA-approved therapies for patient treatment planning.

Abstract

e13612 Background: Identifying targetable genomic alterations to guide treatment options is a key component of personalized cancer care. However, staying up to date with increasing and complex regulatory approvals can be challenging. While Large Language Models (LLMs) offer promise for clinical decision support, they struggle with niche and constantly updating queries like identifying FDA-approved therapies due to insufficient specialized training. Retrieval-augmented generation (RAG) addresses this by dynamically retrieving relevant data from external domain-specific databases to supplement the LLM’s general knowledge without further training. Here, we introduce a RAG-LLM approach that enables accurate and guideline-derived FDA-approved therapy recommendations based on the given patients’ genomic biomarkers, disease type, treatment history, and other clinically relevant information. Methods: To incorporate the latest knowledge about the clinical actionability of genomic biomarkers, we used FDA-approved drug indications from the Molecular Oncology Almanac (MOAlmanac). MOAlmanac contains both unstructured, free-text precision oncology genomic knowledge and structured data fields such as for biomarkers, cancer types, and therapies. We evaluated LLM performance with/without RAG by calculating the proportion of exact matches of correctly predicted therapy recommendations across 234 relationships in 37 cancer types. A paired prompt-answer dataset of 234 short user prompts was manually structured using a shared template. We also used the paired answer dataset and an unstructured dataset with 56 original ‘Indications and Usage’ sections from the FDA drug labels for RAG integration. For all evaluations, we used a general-purpose state-of-the-art Mistral NeMo model. Results: Leveraging structured text and RAG improved model performance by 43% relative to a general-purpose LLM (94% vs. 65.8%), highlighting the benefit of integrating external knowledge for better therapeutic recommendations. Using unstructured text from FDA drug labels with RAG also improved model performance by 10% (72.6% vs. 65.8%), demonstrating RAG’s ability to handle unstructured data. Of note, the RAG-LLM with structured text achieved the highest accuracy with a similar runtime as the general-purpose LLM, suggesting improved accuracy didn’t compromise efficiency. Conclusions: By integrating an expert-curated dataset with RAG, LLMs can accurately suggest evidence-based therapies across various cancer types, which may facilitate precision oncology investigations. Example prompts and responses used in this study are available at llm.moalmanac.org . Approach Accuracy (%) Runtime (min) LLM 65.8 14.43 RAG-LLM (with unstructured text) 72.6 15.09 RAG-LLM (with structured text) 94.0 14.29