Reference-Based Compression of FASTQ Data Using Longest Match Model

This paper proposes a reference-based lossless compression algorithm for FASTQ files using hash table. Firstly, the original FASTQ file is split into B blocks in order to take advantage of multi-thread parallelism. Secondly, each block is divided into three parts: metadata, quality score, and base sequence. After that, we use run-length coding and incremental coding to preprocess the quality score and metadata streams respectively. For the base sequence, we use a carefully designed longest matching model to quickly map it onto the reference sequence so that the longest match of each base sequence in the reference sequence can be obtained, and the match result can be saved for further compression. Finally, the three pre-processed data streams are compressed using algorithms such as zpaq and lpaq9m. Our method is tested against data sets of different read lengths generated on different sequencing platforms, it achieves compression ratio in the range of 0.102-0.229, which is significantly better than the same results from other advanced lossless compression algorithms.