A Robust Log Parsing Algorithm—Practice of Logslaw in Heterogeneous Logs of Pacific Credit Card Center of Bank of Communications(PCCC)

Log is a semi-structured text that records the important behavior of the system operation. Through log analysis, we can understand the system behavior, detect system anomaly, locate system fault and even find the root cause of the fault after a fault is found. Generally, log parsing is the first and crucial step of log analysis. In recent years, a great deal of work has been devoted to log parsing. However, most of the work is based on some strong assumptions. These works can achieve good results in some scenarios, but not in others (e.g., the business log of the bank). What's more, their parsing effect is highly dependent on tokenization. In short, these algorithms are weak in robustness, so our paper proposed a robust log parsing algorithm - LogSlaw. LogSlaw is an online log parsing algorithm clustering logs based on the improved Jaccard Similarity, which can perform log parsing on heterogeneous logs with a large amount of data. To improve the accuracy and efficiency of clustering, we proposed to utilize the pseudo log template and the length difference to early grouping and early abandonment. Our algorithm has high parsing efficiency and effectiveness. It can achieve more than four times the efficiency of existing algorithms and get the best results in most datasets. LogSlaw also has strong robustness. It can achieve good results with a very simple tokenization strategy no matter how complex the scene is. We conducted experiments on both the public datasets and the business log datasets from Pacific Credit Card Center of Bank of Communications (PCCC), and achieved SOTA results on both datasets.