AN OPTIMIZED TWO-TIER INTRUSION DETECTION SYSTEM USING BAYESIAN AND HYPERBAND-TUNED XGBOOST
Girraj Khandelwal, Gajanand Sharma, Brij Bhooshan Gupta
Abstract: Traditional intrusion detection systems relatively rigid detection mechanisms are increasingly insufficient against sophisticated cyberattacks as modern network infrastructures grow in complexity. In this work a hybrid, optimization-driven IDS framework to be induce that integrates feature engineering, class-imbalance handling, Bayesian and Hyperband-tuned XGBoost classifiers, hybrid ensemble learning, and SHAP-based explainable artificial intelligence for cyberattack detection. The proposed framework adopts a two-tier architecture in which the first tier performs binary classification to separate normal from malicious traffic, while the second tier performs multi-class classification of the malicious traffic identified by the first tier into specific attack categories. To improve detection performance and computational efficiency, the framework applies leakage-feature elimination and correlation-based feature reduction, followed by class balancing, prior to model training. Hybrid ensemble models based on stacking and soft voting over the tuned base classifiers are used to improve classification robustness, and SHAP-based explainability is incorporated to support interpretation of the features driving each classification decision. Experimental evaluation shows that the proposed model achieves an accuracy of 99.81%, a precision of 96.36%, a recall of 94.32%, an F1-score of 95.27%, and an area under the ROC curve in the neighborhood of 0.999. These results indicate that the proposed two-tier, dual-optimization IDS framework improves detection robustness relative to conventional single-tier machine-learning baselines evaluated on the same data, while maintaining explainability through SHAP-based analysis.
Keywords: Intrusion detection system; Two-tier classification; XGBoost; Bayesian optimization; Hyperband; Hyperparameter optimization; Ensemble learning; SHAP; Network security
DOI: 10.24874/PES08.03.035
Recieved: 17.06.2026 Revised: 06.08.2026 Accepted: 28.08.2026
UDC:
Reads: 10 
