Automated Log-Based Anomaly Detection under Noise Circumstances

As log data proliferates, it presents opportunities to enhance service reliability and system performance, but also poses challenges. Conventional log analysis methods, reliant on domain knowledge and manual intervention, are inadequate in exploiting this data. Real-world systems encounter noise disruptions, such as disordered or missing logs, further complicating the analysis. To address these issues, we propose LogAR, an innovative model for log-based anomaly detection. By incorporating recurrent neural networks and word embeddings, LogAR’s Auto-Encoder adeptly handles noise and captures the semantic complexities of log sequences. It combines an encoder, decoder, and binary classifier to extract features, reconstruct log entries, and assign normal or abnormal labels. LogAR reduces reliance on domain knowledge and manual intervention, offering a sophisticated tool for managing large-scale systems. Experimental evaluations on the publicly available Hadoop Distributed File System (HDFS) log dataset demonstrate LogAR’s superiority over benchmark methods in anomaly detection. Notably, it exhibits robustness to artificially introduced noise, outperforming the conventional Long-Short Term Memory (LSTM) model. These results underscore LogAR’s effectiveness and adaptability in real-world scenarios.