Deep Reference Frame Interpolation based Inter Prediction Enhancement for Versatile Video Coding

In video coding, bi-directional inter prediction aims to remove temporal redundancy via two-sided previously coded frames as reference. High-quality reference frames are essential to reduce the prediction residuals and improve coding efficiency performance. In this paper, we propose a deep learning-based reference frame interpolation method to enhance bi-prediction by introducing a synthetic frame to reference picture lists. Specifically, reconstructed frames are fed into a well-designed interpolation and filtering network to synthesize a picture which can be regarded as an additional reference of to-be-coded frame. Then, the picture is inserted at the appropriate place in reference picture lists to provide a more reliable reference for subsequent motion estimation and motion compensation. Experimental results show that the proposed method achieves 2.03%/6.96%/6.40% coding efficiency improvements for Y/U/V components under random access configuration, when compared with the VVC reference software VTM-15.0.

I. Introduction

Over the past decades, several video coding standards have been developed to overcome the storage and transmission problem caused by dramatically growing video data. In 2020, the latest Versatile Video Coding (VVC) [1] was officially released by Joint Video Experts Team (JVET). Compared to the previous High Efficiency Video Coding (HEVC) [2], VVC can save about 50% bit-rate with the adoption of several newly developed coding tools [1].