Research on the application of LLaVA model based on QLoRA fine-tuning in medical teaching

Abstract

The augmented reality large language model medical teaching system (ARLMT) integrates augmented reality (AR)with a medical multimodal large language model (LLaVA) based on specifically designed for biomedical applications(LLaVA-Med), employing Quantized Low-Rank Adaptation (QLoRA) to advance medical education. Deployed on resource-constrained AR devices, such as INMO Air2 AR glasses, ARLMT overlays real-time visual annotations and textual feedback on medical scenarios to create an immersive and interactive learning environment. Key advancements include a 66% reduction in memory footprint (from 15.2 GB to 5.1 GB) through QLoRA, enabling efficient operation without compromising performance, and an average response time of 1.009 seconds across various medical imaging categories, surpassing the GPT-4 baseline in both speed and accuracy. The system achieves 98.3% diagnostic accuracy, demonstrating its reliability in real-time applications. By combining visual and textual elements, ARLMT enhances comprehension of complex medical concepts, providing a scalable, real-time solution that bridges technological innovation and pedagogical needs in medical training.