Adaptive graph signal processing for robust multimodal fusion with dynamic semantic alignment

Abstract

Abstract In this paper, we introduce an Adaptive Graph Signal Processing with Dynamic Semantic Alignment (AGSP-DSA) framework to perform robust multimodal data fusion across heterogeneous sources, including text, audio, and images. The proposed approach uses a dual-graph construction to learn both intra-model and inter-modal relations, spectral graph filtering to enhance informative signals, and effective node embeddings via Multi-scale Graph Convolutional Networks. In the semantic-aware attention mechanism, each modality may dynamically contribute to the context with respect to contextual relevance. The experimental outcomes on three benchmark datasets, including Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity dataset, Audio-Visual Event dataset, and MultiModal Internet Movie Database dataset, show that Adaptive Graph Signal Processing with Dynamic Semantic Alignment performs as the state of the art. More precisely, it achieves 95.3% accuracy, 93.6% F1 (Harmonic Mean of Precision and Recall) score, and 92.4% mean average precision on the Carnegie Mellon University Multimodal Opinion Sentiment and Emotion Intensity dataset, improving the MultiModal Graph Neural Network by 2.6% in accuracy. It gets 93.4% accuracy and 91.1% F1 score on Audio-Visual Event dataset, and 91.8% accuracy and 88.6% F1 score on MultiModal Internet Movie Database dataset, which demonstrates good generalization and robustness in the missing modality setting. These findings verify the efficiency of the proposed AGSP-DSA in promoting multimodal learning in sentiment analysis, event recognition, and multimedia classification.