Automated AI-based detection of MRI disease activity in multiple sclerosis: comparison with an expert-adjudicated AI-assisted workflow

Abstract

Abstract Automated magnetic resonance imaging (MRI) analysis is increasingly used in multiple sclerosis (MS) for cross-sectional quantification of lesion burden and brain structures, as well as for longitudinal detection of new or enlarging lesions; however, its performance in real-world workflows remains incompletely understood. We evaluated agreement between fully automated artificial intelligence (AI)-based detection of MRI disease activity and neuroradiologist-adjudicated AI-assisted assessment in patients with relapsing-remitting MS. In this prospective single-center study, standardized 3T brain MRI was performed at 6-month intervals. Overall, 474 MRI examinations, including 343 follow-up scans from 131 patients, were analyzed. AI-only assessment flagged MRI disease activity more frequently than AI-assisted assessment (11.95% vs. 6.12%; OR 2.01, 95% CI 1.40–2.89), with high overall agreement (Gwet’s AC1 0.90, 95% CI 0.86–0.94). Positive agreement was 54.84% (95% CI 38.42–71.26), whereas negative agreement was 95.51% (95% CI 93.67–97.36). AI-only assessment flagged more new lesions per scan (IRR 2.39, 95% CI 1.36–4.19). Among the evaluated AI-derived lesion volume measures, none was associated with discordance. The reported agreement metrics reflect concordance within an AI-assisted workflow rather than independent validation of AI accuracy. Expert interpretation remains essential for borderline or potentially false-positive AI-flagged findings.