Architecture for IDS Log Processing using Spark Streaming

In a large network enterprise system, the use network intrusion detection system (N-IDS) become popular since it has very important role and a challenging task to the network manager in term of security management. Existing network systems develop and expand both in terms of network size, load, and application traffic so the processing of a single IDS is not enough and imposed a high overload on the system. Therefore, there is a need for upgrading a novel IDS system to adapt to the new challenges. To improve the performance of the entire N-IDS system, the traditional way is to replace it with a higher performance server to meet the requirements of processing and storage or using several N-IDS systems. However, in those types of systems, the cost is often expensive but the processing and representation data real-time is still very limited and it does not meet the urgent requirements of security manager. In this paper, we propose novel architecture of distributed log processing and storage tools to improve N-IDS data processing. Our goal is to improve overall system performance and cost-efficient. In this paper, we recommend the use of distributed processing and storage tools to improve N-IDS data processing by Apache Spark and make a comparison with previous works using Hadoop Cluster. Our proposed model introduces a real-time data streaming tool, e.g., Apache Spark Streaming, for near real-time analysis of log processing.