Towards Lightweight Deep Reference Frame for Versatile Video Coding

Deep neural network (DNN)-based methods have demonstrated enormous potential for Versatile Video Coding (VVC) inter prediction enhancement. However, due to their typically high computational complexity, implementing them in practical applications can be challenging. In this paper, we propose a lightweight deep reference frame interpolation network to enhance bi-prediction with low complexity. Specifically, given a pair of bi-directional reconstructed frames, first, we down-sample input frames to reduce the complexity before feeding them into the optical flow estimation network. Then the optical flows are utilized to warp extracted features at three different levels. The warped features are fused to generate the output intermediate frame. The additional reference frame is inserted into the reference picture lists to provide an additional reliable reference candidate. In contrast to previous efforts, the proposed method aims at achieving the trade-off between performance and complexity while maintaining a complexity of about 64 kMACs/pix. Experimental results demonstrate that our method achieves 1.82%/2.43%/2.02% coding efficiency improvements for Y/U/V components under random access (RA) configuration compared to the latest NNVC standard software VTM-11.0_NNVC-5.0.

I. INTRODUCTION

With the rapid advancement of multimedia technology, the demand for storing and transmitting videos is constantly increasing. This has driven the demand for more efficient video coding techniques. The latest version of VVC [1] was officially released by the Joint Video Experts Team (JVET) in 2020, achieving a significant 50% reduction in bit-rate compared to its predecessor [2], while maintaining the same quality of decoded videos. Meanwhile, the recent rapid development of deep learning has had a tremendous impact on video coding. Some of these methods have also been actively researched in both academia and the standardization community [3]–[11]. Deep learning-based methods are usually integrated into the hybrid video coding framework to improve the coding performance of each particular module, such as intra prediction [12], [13], inter prediction [14]–[16], in-loop filter [17], [18], post-processing [19], and rate control [20].