Network Intrusion Detection System Employing Big Data and Intelligent Learning Methods

With the steadily increasing use of computer networks and the exponential growth of data generated from multiple sources, several other practitioners of security management solutions have been faced with the principal objective of providing privacy and security for big data. As a consequence, methods and techniques have been developed for learning to cope with ever-increasing data volumes. Network intrusion detection system is a key component of cyber security systems and is widely used. Big data has posed many challenges for selecting features that have not been addressed by conventional artificial Intelligence and machine learning practices. To overcome this challenge, it is necessary to improve feature selection to present more effective methods to cope with extremely high dimensional space in big data. To address the challenges associated with feature selection in big data analytics and to foster investigations in this clan, we present an analysis of the feature selection challenges encountered by various researchers over the last decade. We propose a system for an intelligent NIDS feature selection model that allows its use as a big data engine. Intelligent NIDS features are selected using a combination of deep learning models and big data methods. Several tools are suggested for relevant data streaming and sustained storage systems in a distributed environment, which include Kafka and Hadoop (HDFS). The Intelligent NIDS Framework experimental procedure comprises a system with normal and malicious traffic, and the NIDS model would perform on cluster frameworks such as the Apache Stream Processing Cluster and the Apache Hadoop Cluster.