Prediction, syntax and semantic grounding in the brain and large language models

Abstract

Abstract Language comprehension involves continuous anticipation of upcoming linguistic input, requiring the rapid integration of syntactic structure and semantic information. To capture the spatio-temporal dynamics of such anticipatory processes during naturalistic language comprehension, we combined electroencephalography (EEG) and magnetoencephalography (MEG), leveraging their complementary sensitivities and high temporal resolution. Using this combined EEG-MEG approach, we investigated word-class-specific neural responses during continuous speech perception and related these findings to word class-level predictability and representational structure in a large language model. Twenty-nine healthy participants listened to a German audio book while their neural responses were recorded. Event-related fields and event-related potentials for different word classes showed highly reproducible, characteristic spatio-temporal signatures, including significant pre-onset activity for nouns, suggesting enhanced anticipatory processing of this word class. Source-space analyses revealed activity patterns extending beyond temporal regions into areas compatible with sensorimotor cortices, suggesting a deeper semantic grounding of nouns in e.g. sensory experiences than verbs. By analyzing word class-specific predictability and representational structure in the transformer-based language model Llama, we provide a computational reference frame that complements the neural findings at the level of word classes. These findings highlight the power of simultaneous MEG-EEG recordings in unraveling the predictive, syntactic, and semantic mechanisms that underlie language comprehension.