The order of multisensory associative sequences is reinstated as context feature during successful recognition

Abstract

Abstract The ability of the human brain to encode and recognize sequential information from different sensory modalities is key to memory formation. The sequence in which these modalities are presented during encoding critically affects recognition. This study investigates the encoding of sensory modality sequences and its neural impact on recognition using multivariate pattern analysis (MVPA) of oscillatory EEG activity. We examined the reinstatement of multisensory episode-specific sequences in n = 32 participants who encoded sound-image associations (e.g., the image of a ship with the sound of a frog). Images and sounds were natural scenes and 2-second real-life sounds, presented sequentially during encoding. During recognition, stimulus pairs were presented simultaneously, and classification was used to test whether the modality sequence order could be decoded as a contextual feature in memory. Oscillatory results identified a distinct neural signature during successful retrieval, associated with the original modality sequence. Furthermore, MVPA successfully decoded neural patterns of different modality sequences, hinting at specific memory traces. These findings suggest that the sequence in which sensory modalities are encoded forms a neural signature, affecting later recognition. This study provides novel insights into the relationship between modality encoding and recognition, with broad implications for cognitive neuroscience and memory research.