The Study of Feature Engineering in Machine Learning and Deep Learning for Network Intrusion Detection Systems

With the rise of sophisticated cyberattacks, the efficiency of intrusion detection systems becomes paramount. Machine learning (ML) and deep learning (DL) models used for intrusion detection often encounter datasets with irrelevant or redundant features, leading to low performance. To address this challenge, feature engineering techniques are important in extracting the most informative features, enabling faster and more accurate detection of malicious patterns. This paper investigates a comparative analysis of four feature engineering methods using a historical archival dataset: entropy, mutual information, chi- squared statistics, and ANOVA. By evaluating and comparing the effectiveness of these methods under different conditions of ML/DL models, this study aims to provide insights into their respective strengths and weaknesses, guiding the selection of the most suitable approach to improve the efficiency of network intrusion detection systems.