Emotion Recognition with Deep Learning

Abstract

This paper presents a unified Flask-based web application for multimodal emotion recognition, integrating speech, facial expression, and text analysis into a single framework. The system employs state-of-the-art deep learning and transformer-based models to enable accurate, real-time emotion detection across diverse input modalities. For audio-based recognition, CNN–LSTM architecture is used to perform multilingual speech emotion analysis, outperforming traditional approaches that rely on MFCC and chroma features. The facial expression module leverages a Mini-Xception convolutional neural network, which is robust to variations in lighting, pose, and occlusion; it also incorporates a temporal smoothing mechanism to stabilize predictions in live video streams. The text analysis module utilizes a DistilRoBERTa transformer to interpret nuanced emotional cues in written input. All three modalities are integrated into the Flask web interface supporting webcam video streaming, audio recording, and text entry. User data management and authentication are handled securely via a MySQL database. By combining convolutional and transformer architectures in a multimodal fusion framework, the proposed system achieves enhanced accuracy, adaptability, and consistency across modalities. Potential applications include affective computing, mental health monitoring, virtual assistants, and empathetic human–computer interaction, thereby improving the emotional responsiveness of automated systems.