Event detection via graph convolutional networks for multi-source multimodal integration

Abstract

In real-time event detection, social media platforms like Twitter, Instagram, and Weibo provide valuable data, where users share updates, opinions, and multimedia content. However, existing event detection algorithms typically rely on a single data source or modality, limiting their ability to process the complex, multimodal nature of event information. To address this, we propose a novel framework that integrates multi-source, multimodal data to improve event detection robustness. Using graph convolutional networks, the framework captures both textual and visual information, incorporating syntactic and dependency details. It employs cross-modal attention to combine features from text and images, enhancing the system’s understanding of how these modalities complement each other. The model identifies event trigger words in the text and classifies events based on both textual and visual cues. Our approach improves event detection accuracy and reliability. Evaluation using news data from the same period as social media posts shows that combining text, images, and news data increases event detection and classification accuracy, offering a more dynamic and comprehensive solution.