Learning-Based Fast Splitting and Directional Mode Decision for VVC Intra Prediction

As the latest video coding standard, Versatile Video Coding (VVC) is highly efficient at the cost of very high coding complexity, which seriously hinders its practical application. Therefore, it is very crucial to improve its coding speed. In this paper, we propose a learning-based fast split mode (SM) and directional mode (DM) decision algorithm for VVC intra prediction using a deep learning approach. Specifically, given the observation that the SM distributions of coding units (CUs) of different sizes are significantly distinct, we first design the neural networks separately and train the SM models for all CUs of different sizes to obtain the probability of SMs and skip the unlikely ones. Second, given a similar observation that the DM distributions of CUs of different sizes are distinct, we design neural networks to train the DM models for all CUs of different sizes separately to obtain the probabilities of DMs, and then adaptively select candidate DMs based on probabilities of their located SMs. Third, after an SM is checked, we select its probability, residual coefficients, rate-distortion (RD) cost, etc. as features, and design a lightweight neural network (LNN) model to early terminate SM selection. Experimental results demonstrate that the proposed algorithm can reduce the encoding time of VVC by 70.73% with 2.44% increase in Bjøntegaard delta bit-rate (BDBR) on average.

I. Introduction

With the rapid development of information technology, various video applications have been widely used in our daily lives, such as network broadcasting, video conferencing and smartphone communications. Meanwhile, users are demanding higher video quality and various ultra-high definition (UHD) video applications are becoming popular, such as 4K/8K video and virtual reality (VR) video, causing the explosive growth of visual data. The previous generation of coding standard, namely High Efficiency Video Coding (HEVC) [1], has gradually failed to meet the market demand for more efficient coding efficiency targets for high-resolution video. As a result, the Joint Video Exploration Team (JVET) has developed the latest generation of video coding standard, i.e., Versatile Video Coding (VVC) [2]. Thanks to a range of new tools, such as the quad-tree plus multi-type tree (QTMT) structure of coding unit (CU) partitioning and the additional intra prediction mode (IPM), VVC achieves significantly higher coding efficiency compared with HEVC [3]. However, its computational complexity is also greatly increased, making VVC unsuitable for real-time applications [4].