AOCL-Compression - A High Performance Optimized Lossless Data Compression Library

Data compression is the process of encoding (or compressing) information using fewer bits than originally present in the data stream or signal. Depending upon whether this process is invertible or not, data compression can be lossless or lossy. While there exists various popular implementations of different lossless data compression methods, they are unable to completely meet the performance requirements demanded by the ever-increasing data usage of the applications. In this paper, we present a comparative survey of different loss less compression methods and introduce a high performance compression library called AOCL-Compression optimized for x86 architecture in general and AMD's “Zen”-based processors in particular. This library supports LZ4, LZ4HC, ZLIB, ZSTD, LZMA, BZIP2, and Snappy based compression methods. This paper discusses the design features of the new library framework and the algorithmic optimizations implemented for the different compression methods. Results highlighting the performance benefits of this new library implementation over the reference implementations of the respective compression methods are also presented.