Towards Massive Log Anomaly Detection Based on an Enhanced Multi-dimensional Time-domain LSTM

Malicious users pose a significant threat to the security of sensitive user data. Web logs contain valuable implicit information about user behavior, making it crucial to dynamically detect and identify abnormal user behaviors through log data for network security. However, most current log-based anomaly detection methods only use log event indexes parsed from the log data, ignoring the association information of log items across multiple dimensions and long-time domains. In this paper, we propose a dynamic anomaly detection framework based on an enhanced multi-dimensional time-domain LSTM algorithm, which analyzes and models user behaviors from massive log data for achieving real-time detection and identification of abnormal behaviors. Firstly, we provide a unified definition of user behavior to solve the polysemy problem. Then, based on the time series data, we predict the user behavior trajectories and detect anomalous behaviors using the proposed model. We also trace back and restore the identified anomalies to analyze the causes. Finally, we evaluate the proposed method on two log datasets obtained from a real online system. Experimental results demonstrate that our method significantly outperforms several state-of-the-art methods.