Enhancing resident learning in anticoagulation management: A single-institution quality improvement study with comparison to large language models.

Abstract

e23308 Background: Anticoagulation management is complicated by evolving guidelines and patient-specific factors, particularly in patients with cancer, pregnancy, and renal or hepatic dysfunction. This quality improvement initiative evaluated the impact of a structured educational intervention on resident anticoagulation knowledge and contextualized trainee performance using large language models (LLMs). Methods: We conducted a single-center quality improvement educational intervention involving Internal Medicine residents (PGY-1 to PGY-3). A 10-item multiple-choice questionnaire assessing anticoagulation stewardship in special populations was administered via REDCap before and after a guideline-based teaching session. Mean percent correct scores were calculated overall and for a prespecified subset of cancer-associated thrombosis questions. For comparison, two LLMs, ChatGPT and OpenEvidence, independently completed the same questionnaire before and after exposure to the educational material. Descriptive statistics were used. Results: 42 residents completed both assessments. Mean scores improved from 42.63% pre-intervention to 74.52% post-intervention, representing a 31.89 percentage point increase. Performance on cancer-associated thrombosis questions improved from 50% to 74.28%. ChatGPT and OpenEvidence each scored 80% on both pre- and post-intervention assessments, with no change in responses following exposure to the educational material. Incorrect responses did not overlap between the two LLMs. Both models provided reasoning, though only OpenEvidence cited peer-reviewed references. Conclusions: This intervention demonstrates that targeted, guideline-based anticoagulation education can produce clinically meaningful improvements in trainee knowledge, particularly in complex domains such as cancer-associated thrombosis. In contrast, the static performance of LLMs highlights their role as adjunctive tools rather than adaptive learners, underscoring the importance of structured medical education for clinical competency development. Pre- and post-intervention questionnaire results and comparison to LLMs. Pre-intervention % correct Post intervention % correct Absolute % Increase Answered correctly by ChatGPT Answered correctly by OpenEvidence 1. Provoked DVT 26.2 66.7 40.5 Yes Yes 2. Provoked PE 14.3 64.3 50.0 Yes Yes 3. ESRD, diagnosis of DVT 66.7 73.8 7.1 Yes No 4. Thrombophilia testing 26.2 83.3 57.1 Yes Yes 5. 3 rd trimester pregnancy, diagnosis of DVT 42.9 85.7 42.9 Yes Yes 6. Lung Cancer, eGFR > 60, diagnosis of DVT 64.3 92.9 28.6 Yes Yes 7. RCC, eGFR <30, diagnosed with VTE 28.6 47.6 19.0 No Yes 8. GI cancer, diagnosed with VTE 21.4 73.8 52.4 Yes No 9. HCC, VTE prophylaxis 52.4 61.9 9.5 No Yes 10. Breast Cancer, ambulatory, VTE prophylaxis 83.3 95.2 11.9 Yes Yes