Virtual oncology collaborative tumor board using multiple artificial intelligence agents.

Abstract

1563 Background: Clinical guidelines are complex documents with tables and figures. Finding specific answers to questions within these guidelines can be challenging and time-consuming. Current AI tools often struggle to extract information from these PDF guidelines. To address this, we developed a novel system that uses a group of artificial intelligence (AI) agents, where each agent is designed to perform a distinct job, like finding information, reading documents, or summarizing findings. These agents communicate with each other to analyze complex clinical guidelines, working as a team to answer physician questions like trained oncologists discussing cases in a tumor board environment. Methods: Publicly available PDF guidelines published by the ASCO from Jan 2021 to Dec 2024 were acquired. A three-agent framework was constructed using the AutoGen platform, comprising a Coordinator Agent, a PDF Viewer Agent, and a Reviewer Agent. The Coordinator Agent selects the appropriate guideline based on a user's question; the PDF Viewer Agent extracts information from the selected guideline file, and the Reviewer Agent generates a summary of the findings answering the original question. The agents were powered by Anthropic’s Claude 3.5 Sonnet. The primary objective of the study was to evaluate the platform’s accuracy in selecting the relevant guideline based on user questions and in subsequently answering those questions accurately. Results: A total of 34 ASCO guidelines were obtained, covering a range of cancer types: breast (15), GI (4), head and neck (4), thoracic (4), neuro-oncology (3), GU (2), melanoma (1), and gynecologic (1). One hundred question-answer pairs were created by board-certified oncologists based on these guidelines to evaluate the system’s performance. It's important to note that these answers were based directly on the information in the guidelines and may not always reflect the most current clinical knowledge, thus serving as a rigorous test of the framework’s ability to adhere to the provided documents. Our multi-agent framework achieved a 93% accuracy rate in matching user questions with the correct guideline and answered 88% of the questions accurately. Comparatively, when the same questions were evaluated using OpenAI’s GPT-4o (ChatGPT) and Claude 3.5 Sonnet without the multi-agent framework, the accuracy was significantly lower at 48% and 49%, respectively. The total computational cost of processing all questions using the multi-agent framework was 13.44 USD. The complete code, dataset, and detailed results are publicly accessible at https://github.com/jwang-580/ASCO_guideline_agents. Conclusions: This study demonstrates that a collaborative AI agent system can accurately provide answers from clinical guidelines that is more accurate than ChatGPT and similar software. Our results suggest a promising way to develop more effective AI tools for clinicians to use in their practice.