Design and Development of Scalable SIEM as a Service using Spark and Anomaly Detection

Existing Security Incident and Event Management (SIEM) systems traditionally gather data from various log files produced by applications and network security systems like firewalls and IDS/IPS. Modern SIEMs now integrate machine learning capabilities to analyze log file patterns, learning from them to detect and potentially prevent similar attacks while adapting to evolving threats. However, deploying AI-driven SIEM solutions on the cloud to serve multiple clients still presents challenges due to the diverse and voluminous nature of log files, leading to significant preprocessing costs for managing various file formats and unpredictable throughput. In this paper, we proposed a dynamic and efficient SIEM as a service model called SIEMAS integrating Random Forest (RF) as the core detection engine. Additionally, we introduced log file normalization based on Amazon Kinesis and utilized Apache Spark to streamline this normalization process. Our detection approach uses anomaly detection to instantly recognize patterns of malicious events and respond to them by leveraging the RF algorithm. Finally, we have developed a prototype as a proof of concept of the system functionality and conducted experiments to demonstrate its detection accuracy and performance, which have shown to be efficient in practical scenarios. The results showed that our proposed solution achieves an overall detection accuracy of 98% while achieving a 33 % higher speed of detection than without using distributed computing.