Semantic congruence impacts audiovisual processing in the Colavita effect

Abstract

The Colavita effect is a phenomenon that occurs when participants are presented with auditory, visual, and audiovisual stimuli and are tasked to identify the modality of the stimuli. When audiovisual stimuli are presented, participants occasionally miss the auditory component and report that the stimulus was visual only – reflecting visual dominance. The Colavita effect has been shown to be resistant to manipulations of semantic correspondence, which is surprising considering that semantic information impacts multiple perceptual tasks. It is possible that the brief auditory stimuli used for Colavita tasks do not allow for recognition, preventing the recruitment of semantics. We first replicated the Colavita effect using words as stimuli. We then manipulated semantic correspondence by pairing the auditory (sounds or words) and visual (pictures) components of the stimuli either with the same concept (e.g., the picture of a cat with the sound of a cat or the word ‘cat’) or different concepts (e.g., the picture of a cat with the sound of a dog or the word ‘dog’). In Experiment 2, we observed that semantic correspondence influenced the magnitude of the Colavita effect: when the visual and auditory component corresponded to different concepts, the size of the Colavita effect was reduced. Finally, in Experiment 3 we observed that using visual and auditory stimuli from different categories further reduced the effect. We suggest semantic congruence influences the unity of bimodal stimuli: congruent stimuli are more easily perceived as a single entity while incongruent stimuli are not as easily perceived as a single entity, leading to a reduction of ‘visual-only’ responses.