Anomaly Detection Model for Log Based on LSTM Network and Variational Autoencoder

Log-based anomaly detection is the key to ensuring that today's large-scale distributed systems can function properly. Analyzing log data that records the running status of software systems and quickly and accurately identifying abnormal parts is the primary task of maintaining system and software security and stability. The main contribution of this paper is the presentation of a new model combining LSTM network and variational autoencoder model: LogLVAE. The model includes a data preprocessing phase and an anomaly detection phase. In the data preprocessing phase, the current most efficient online log parsing algorithm, Drain, is used to parse the log data into log templates, and then the log template sequence is divided by the sliding window algorithm. In the anomaly detection phase, the model first uses LSTM to be able to extract semantic features of log sequences by contextual information, then reconstructs the input data by VAE, calculates the reconstruction error using the reconstructed data and the input data, and detects anomalies in log sequences and reports the results to the administrator if the reconstruction error exceeds a predefined threshold. The results of our experiments, which were based on two real log datasets, indicate that LogLVAE outperformed other algorithms in terms of overall performance.