Learning-Based Anomaly Detection Using Log Files with Sequential Relationships

Modern IT systems have been transitioning from traditional on-premises solutions to a dynamic mixture of on-premises and off-premises solutions. This transition has also included a trend to run more systems on software-defined resources. The ease of setting up new software-defined servers and systems has led to an increase in IT system complexity as well as the amount of log data generated. Automatic log analysis has become a subject of interest because of the problems with manual log analysis in case of intrusion detection and root-cause analysis. Therefore, this paper proposes and tests a sequence based anomaly detection method. The work has been done in collaboration with the Swedish Social Insurance Agency’s IT department. Real system log data with high privacy requirements and limited available information was generated for training and testing. The generated log data was produced with expected time regions of anomalous behavior. Our proposed anomaly detection model was then able to perform at a state-of-the-art level and could accurately detect certain error types. Showing the potential of the approach when applied directly to a real-world system.