Log-Based Anomaly Detection with Transformers Pre-Trained on Large-Scale Unlabeled Data

It is crucial to automatically detect anomalous patterns in system logs to protect computer systems from cyber attacks and malfunctions. However, as log data is becoming increasingly complex and labeled logs are difficult to obtain, it poses serious challenges to existing methods. To this end, this paper introduces the pre-training and fine-tuning paradigm to the log analysis domain and proposes a novel log anomaly detection framework. We propose the masked log reconstruction approach to pre-train a Transformer-based foundation model and fine-tune it for the event prediction task to obtain the anomaly detector. Our training methods exploit the sequential information within unlabeled logs with self-supervised learning. Experimental results on two public datasets demonstrate the performance superiority of our framework compared with existing state-of-the-art methods. More importantly, it is suitable for real-world scenarios where labeled logs are difficult to acquire.