TurboLog: A Turbocharged Lossless Compression Method for System Logs via Transformer

System logs provide valuable information for intrusion detection and forensic analysis. To counter cyber attacks, enterprises widely deploy monitoring software and record fine-grained system events within the operating system through system logs. However, system logs can pile up in large quantities as the complex nature of modern computer systems and the covert nature of cyber attacks. Storing system logs efficiently is an important and challenging task. Lossless compression techniques provide an intuitive idea of reducing the size of system logs. Recently, researchers have applied Deep Neural Networks (DNNs) in designing compression strategies, achieving remarkable outcomes. Unfortunately, general-purpose lossless compression methods fail to achieve effective compression due to their inadequate extraction of structural redundancy from system logs. Certain methods opt to add a few preprocessing steps before log compression to enhance compression efficiency. Nevertheless, their RNN-based models face challenges in efficiently utilizing GPU resources and can not handle parallel computation. Therefore, their methods still demand a significant compression time. In this paper, we propose TurboLog, a novel transformer-based compression method specifically designed for system logs. To address the above deficiencies, TurboLog initiates by executing a series of preprocessing steps to reduce the structural redundancy and numerical redundancy within raw logs. Subsequently, Tur-boLog uses a transformer-based probability estimator to model log data in parallel. Finally, TurboLog combines the probability estimation of log data with arithmetic coding to accomplish log compression. We use a large public dataset to evaluate TurboLog. The results show that TurboLog can achieve the highest compression ratio of 65.25×. Furthermore, compared to the state-of-the-art method, TurboLog decreases compression time by 75% while concurrently enhancing the compression ratio by 20%. In conclusion, TurboLog achieves much better space savings than existing methods and points an optional direction for future research.