Semantic enhancement and cross-modal interaction fusion for sentiment analysis in social media

Abstract

The rapid development of social media has significantly impacted sentiment analysis, essential for understanding public opinion and predicting social trends. However, modality fusion in sentiment analysis can introduce a lot of noise because of the differences in semantic representations among various modalities, ultimately impacting the accuracy of classification results. Thus, this paper presents a Semantic Enhancement and Cross-Modal Interaction Fusion (SECIF) model for sentiment analysis to address these issues. Firstly, BERT and ResNet extract feature representations from text and images. Secondly, the GMHA mechanism is proposed to aggregate important semantic information and mitigate the influence of noise. Then, an ICN module is created to capture complex contextual dependencies and enhance the capability of text feature representations. Finally, a cross-modal interaction fusion module is implemented. Text features are considered primary, and image features are auxiliary, enabling the profound integration of textual and visual features. The model's performance is optimized by combining cross-entropy and KL divergence losses. The experiments are conducted using a dataset collected from public opinion events on Sina Weibo. The results demonstrate that the proposed model outperforms comparison models. The SECIF model improves by 11.19%, 82.27%, and 4.83% over the average accuracy of the text-only, image-only, and multimodal models, respectively. The proposed SECIF model is compared with ten baseline models on the publicly available datasets. The experimental results show that the SECIF model improves accuracy by 4.70% and F1 score by 6.56%. Through multimodal sentiment analysis, governments can better understand public emotions and opinion trends, facilitating more targeted and effective management strategies.