Kitsune Dataset Analysis via BigData and Deep Learning Techniques

During the comprehensive digitalization of information processes, the amount of network traffic is growing rapidly, which leads to an increase in the frequency and complexity of network threats. Thus, developing effective network intrusion detection systems (NIDSs) that can detect complex multi-step attacks in a timely and accurate manner is one of the important challenges. In this context, BigData and Machine Learning technologies can greatly improve the accuracy and timeliness of complex attack detection. The paper presents NIDS, which consists of two machine learning models with LSTM layers for analyzing long and short sequences and is capable of processing large amounts of data through the use of big data processing techniques. The main goal of this study is to demonstrate the proposed architecture, as well as validate its characteristics by testing the Short-Term information module on an existing data set. The used machine learning models were trained on the Kitsune dataset on a binary classification task and performed in 93% accuracy and 0.03 losses respectively.