Clustering Enabled Robust Intrusion Detection System for Big Data Using Hadoop–PySpark

In the era of burgeoning networks and the proliferation of novel devices, safeguarding computer security presents an ever-growing challenge. The increasing attack surface grants cybercriminals ample opportunities to exploit vulnerabilities within systems. While the internet fosters innovation across diverse sectors, it also exposes us to perils in the form of cyber attacks and nefarious activities. Consequently, the necessity to robustly identify and counteract these threats becomes paramount. Network Intrusion Detection Systems (NIDS) have a pivotal role in upholding system confidentiality, availability, and integrity by actively monitoring and responding to potential breaches. This study addresses the complex predicaments arising from managing copious data volumes and imbalanced class distributions in intrusion detection, a result of the exponential escalation of network traffic. A hybrid framework is proposed to bolster the accuracy of identifying minority classes within imbalanced datasets. This innovative hybrid framework combines K-means clustering and the Random Forest classifier, custom-crafted for big data processing through the utilization of Hadoop with PySpark. The proposed model shows its efficacy while evaluating training model by the NSL- KDD testing dataset, applying accuracy assessment as well as processing time evaluation. The outcomes of the proposed model, are measured with an accuracy, a precision, a recall, and an F1-score which are 0.9987, 0.9994, 0.9991, and 0.9986 respectively. These results outperforms existing established models in the detection of intrusions.