A hybrid transformer-BiLSTM model optimized with Firefly Algorithm for network traffic anomaly detection

Abstract

Network Traffic Anomaly Detection (NTAD) is essential for proactive cyber defense against increasingly sophisticated threats. This paper presents a data-driven framework that integrates adaptive signal decomposition, a hybrid attention-recurrent architecture, and metaheuristic optimization for timely anomaly prediction. Raw traffic sequences are first preprocessed via Empirical Mode Decomposition (EMD) to mitigate non-stationarity and suppress noise, yielding denoised intrinsic mode functions. The refined signal is then modeled by a hybrid deep network that couples a multi-head self-attention mechanism—capturing global, long-range dependencies—with a Bidirectional Long Short-Term Memory (BiLSTM) network that encodes bidirectional temporal dynamics. To circumvent the sensitivity of deep models to hyperparameter selection, the Firefly Algorithm (FA) is employed for automated, population-based optimization. Extensive evaluations on benchmark datasets demonstrate that the proposed EMD-FA-Transformer-BiLSTM model attains state-of-the-art performance, outperforms baseline and state-of-the-art models across all evaluated metrics, with statistically significant improvements in both regression error and classification F 1 -score.