Using Transformer Models and Textual Analysis for Log Parsing

Log parsing has become an essential tool for extracting valuable information from a vast volume of log lines. It involves identifying standard patterns and extracting templates from these lines, enabling researchers and companies to employ advanced mining techniques like log deduplication and log anomaly detection. However, existing log parsing approaches have limitations. They often operate on small batches of log text and lack consideration for the entire context, necessitating prior knowledge of the log dataset. Furthermore, there is a scarcity of practical experience reports on the utilization of these log parsing approaches in the literature. In our paper, we address these challenges by proposing a novel log parsing approach that combines Transformers with a customized textual analysis. This textual analysis balances the density clustering of similar log lines, the calculation of word frequencies inside each cluster and the presence of the words inside an English vocabulary to parse new log lines. Our method outperforms existing parsing methods in terms of accuracy and operates as an unsupervised learning approach, eliminating the need for prior knowledge. Additionally, we present a proof-of-concept application of our parsing method in an industrial setting, showcasing its practical implementation.