Using large language models to identify advance care planning in oncology patients who present to the emergency department.

Abstract

e13642 Background: Large language models (LLMs) can identify advance care planning (ACP) in unstructured notes in a fraction of the time required by human reviewers. When patients with metastatic cancer and poor prognosis present to the emergency department (ED), LLMs may enhance care by providing a quick and accurate method to identify ACP during a clinical crisis. No prior study has examined the efficiency of LLMs to identify ACP documentation at the time of an ED visit, comparing to real-time manual abstraction. Methods: This is a secondary analysis of a prospectively enrolled sample in a feasibility study for an ED-based, ACP intervention. Patients were English-speaking adults aged 50 and older with metastatic solid tumor cancer and a prognosis less than one year. We applied a previously validated LLM (Agaronnik et al. JPSM 2025, Agaronnik et al. JCO-OP 2025) to notes up to 6 months prior to the ED visit, comparing to real-time manual abstraction by trained research assistants. The LLM required output to include source text to support identification of ACP. Output included a hallucination score, which is a measure of the likelihood of an LLM producing false evidence. Qualitative analysis was used to review output. Results: 1262 patients had available notes for application of the LLM. Of 387 patients who had ACP identified by chart review, the LLM identified 363 patients (93.8%) as having ACP. Of 875 patients who were not identified as having ACP with chart review, the LLM found ACP in 551 patients (63.0%) of these patients. Examples of documentation identified by both the LLM and chart review include goals of care (“understands that her prognosis is poor but she states that she ‘is not ready to die yet’”), limitations on life-sustaining therapy (“she reports she would probably not want CPR but would want a breathing tube if it was temporary”), and hospice (“referral to hospice would be reasonable to facilitate safe discharge, especially in light of recent GOC discussions”). The LLM also identified documentation that was missed by chart review but may be relevant to patient care: “her main concern is for her [family] and 'how they will cope' after her eventual passing” and “I know that I am running out of options, but I'm not ready for hospice yet.” A common reason for false positives was statements in notes about health care proxy forms being available in the clinical chart. Average hallucination index for documentation identified by the LLM was low. LLM often found information missed by human reviewers. LLM extraction of ACP required several seconds per patient compared to 1-4 minutes per patient with chart review. Conclusions: LLMs can optimize review of ACP in the ED for patients admitted with serious illness, identifying ACP in a fraction of the time required by human reviewers. LLMs can be used to facilitate faster shared decision-making and accurate delivery of goal-concordant care in emergency settings.