A Hybrid CNN–BiLSTM Framework with LightGBM Stacking, SDN-Gate, and PSO-Based Threshold Optimization for Enhanced Intrusion Detection in SDN Environments

Received: 2 March 2026 | Revised: 31 March 2026 and 22 April 2026 | Accepted: 23 April 2026 | Online: 6 June 2026

Corresponding author: Maryam Yasmin Ahman

Abstract

Intrusion detection in Software-Defined Networking (SDN) remains challenging due to dynamic control plane traffic and the scarcity of realistic datasets. Conventional Intrusion Detection Systems (IDSs) often struggle to capture diverse SDN-specific threats, including flow rule flooding, topology poisoning, control plane reflection, and other controller-targeted anomalies. This study presents a hybrid Convolutional Neural Network (CNN)–Bidirectional Long Short-Term Memory (BiLSTM) → Light Gradient Boosting Machine (LightGBM) + SDN-Gate with Particle Swarm Optimization (PSO) framework designed to enhance detection accuracy and control plane reliability. A realistic SDN traffic dataset was generated in a GNS3 testbed combining OpenDaylight, Open vSwitch, and multiple Linux hosts, encompassing both general and SDN-specific attacks. The proposed framework employs convolutional and bidirectional recurrent layers for spatial–temporal feature learning, Synthetic Minority Over-sampling Technique–Edited Nearest Neighbor (SMOTE-ENN) for class imbalance mitigation, and LightGBM stacking with PSO-based threshold optimization for calibrated decision fusion. The SDN-Gate, a lightweight LightGBM-based verifier, reevaluates SDN-specific predictions using confidence margins, and verifies and selectively demotes uncertain SDN-specific predictions, thereby reducing false alarms and improving controller-level reliability, and providing a practical foundation for IDS implementations in SDN environments. Experimental results demonstrate 99.80% accuracy and 97.16% Macro-F1 on the full dataset, and 97.22% accuracy and 97.22% Macro-F1 on the Address Resolution Protocol (ARP) and Man-in-the-Middle (MITM) attacks subset, outperforming baseline deep and shallow learning models. Overall, the proposed framework provides a reliable and explainable approach to improving and strengthening the security of SDN networks in real-world settings.