Rate Distortion Optimization of H.264 with Main Profile Compatibility

Using soft decision quantization rather than the conventional hard decision quantization, this paper studies a joint rate distortion design of motion prediction, quantization and entropy coding for the H.264 main profile encoding. Specifically, a soft decision quantization algorithm is proposed based on the context adaptive binary arithmetic coding method in H.264. The proposed algorithm is proved to achieve optimal soft decision quantization for a block with given motion prediction and quantization step size in the sense of minimizing the true rate distortion cost. It is then used in jointly designing motion prediction and residual coding for H.264 main profile coding. Experiments have been conducted based on the reference encoder JM82 of H.264. Comparative studies show that the proposed joint design method achieves an average 10% rate reduction while maintaining the same quality over the rate distortion method in the reference software of H.264

I. Introduction

H.264, the newest hybrid video compression standard [6], has proved its superiority in coding efficiency over its precedents, e.g., it shows a more than 50% rate reduction over the popular MPEG-2. As the enormous volume of video data is constantly fueling the demand for better and better compression [7], it is desirable to study how rate distortion (RD) methods based on fundamental theoretical research in RD theory can be used to further enhance the compression performance in the H.264 compliant coding environment.