Designing JPEG quantization matrix using rate-distortion approach and human visual system model

JPEG is an international standard for still image compression. The JPEG baseline algorithm allows users to supply the custom quantization table and Huffman table to control the compression ratio and the quality of the encoded image. Methods for determining the quantization matrix are usually based on (i) rate-distortion theory and (ii) spatial masking effects of the human visual system. Wu and Gersho (1993) proposed a recursive algorithm for generating picture-adaptive quantization tables based on rate-distortion approach but the complexity of the encoding algorithm is rather high. In this paper, we propose improvements to the Wu-Gersho's algorithm and a new bit allocation algorithm. Simulation results show that our new algorithm is superior to the Wu-Gersho's algorithm in terms of speed and peak signal to noise ratio (PSNR). Moreover, by incorporating the human visual system (HVS), our proposed coder can encode images with better visual quality.