Open-source modular AI coupled with agentic AI for comprehensive breast cancer note generation and guideline-directed treatment comparison.

Abstract

e13685 Background: Generative Artificial Intelligence (GenAI) has demonstrated promise as a clinical decision support tool. Previous studies utilized closed-source large language models (LLMs) such as GPT-4o (via the chatbot ChatGPT) to evaluate GenAI's role in healthcare. However, these LLMs may change, causing challenges with reliability and reproducibility. Hallucinations are especially concerning in healthcare, so methods such as grounding and retrieval augmented generation (RAG) are important tools that may reduce or eliminate hallucinations. Methods: The goal of this study was to enhance GenAI with agentic AI and vector-based RAG, using only open-source tools and LLMs to produce reliable breast cancer summaries and treatment evaluations. A container with Neo4j vector database, LangChain, Docling, and Jupyter was created to review HL7 patient charts containing mCODE data. Ollama was used to pull the LLMs llama3.2, gemma2:2b, qwen2.5, and phi3:mini. Synthetic Breast Cancer Dataset collected from The mCODE Project was collected, and a custom HL7-mCODE module was made to make patient data LLM-ingestible. The workflow was as follows: a modular (i.e., swappable) LLM with RAG would iterate over patient notes to extract all information related to cancer in their chart. A subsequent LLM (i.e., agentic AI) would compare the first AI's extraction with an mCODE summary to evaluate if there were any errors, remove them, and return a corrected cancer history. After this comparison was complete, another AI agent would evaluate for missing oncologic information (such as HER status) and return a list of known and unknown information for breast cancer. For the last step, NCCN Breast Cancer guidelines (Version 6.2024 11-11-2024) were converted to LLM-ingestible text via IBM's docling and placed in a vector database. The last AI agent would compare the patient's cancer details and treatment to compare with the guidelines. Results: 724 patient charts were generated with various modular AIs. No hallucinations were observed in the outputted data (i.e., no fabricated diagnoses, cancer details, treatments, etc.), and no incorrect interpretations were found. Most outputs correctly stated they could not assess NCCN guidelines due to insufficient information in the patient chart; charts with sufficient information to follow a specific guideline returned correct comparisons. In one case, Microsoft's phi3:mini was able to discern that while the guidelines were not followed, the provided guidelines are newer than the date the synthetic patient received treatment. Conclusions: Agentic AI as a utility for grounding, summarizing, and quality assurance demonstrates promise as an augmentation for GenAI to produce effective CDS tools for breast cancer history collection, evaluation, and treatment. Further studies with knowledge graphs may further improve their utility.