Large language model approach to uncover reasoning patterns in forensic psychiatric assessment

Abstract

Abstract Forensic psychiatric assessment (FPA) evaluates whether mental disorders impair an individual’s criminal responsibility, yet the reasoning underlying these high-stakes judgments remains largely implicit and susceptible to subjectivity. We investigated whether large language models (LLMs) can quantify and elucidate reasoning patterns embedded in FPAs, and which components contribute most. We analyzed 253 FPAs conducted at a Finnish forensic hospital between 2018 and 2023. Text from each report section was embedded using a Finnish Sentence-BERT model, and section-specific embeddings were classified with support vector machines within a nested cross-validation framework. Model performance was evaluated using the area under the receiver operating characteristic curve (AUROC), complemented by word- and sentence-level explainability analyses. Among individual sections, the psychiatric evaluation (AUROC 0.90, 95% CI 0.86–0.94), psychological assessment (0.88, 0.84–0.92), and previous records (0.83, 0.78–0.88) were most predictive of criminal responsibility. Combining multiple sections further improved discrimination (0.94, 0.90–0.97). Linguistic markers related to psychosis, disorganization, and psychiatric treatment were associated with criminal irresponsibility, whereas substance use, antisocial traits, and organized behavior were more frequent among those deemed responsible. These findings demonstrate that LLM-based methods can identify and quantify textual patterns associated with forensic psychiatric decision-making.