FastSCCNet: Fast Mode Decision in VVC Screen Content Coding via Fully Convolutional Network

Screen content coding have been supported recently in Versatile Video Coding (VVC) to improve the coding efficiency of screen content videos by adopting new coding modes which are dedicated to screen content video compression. Two new coding modes called Intra Block Copy (IBC) and Palette (PLT) are introduced. However, the flexible quad-tree plus multi-type tree (QTMT) coding structure for coding unit (CU) partitioning in VVC makes the fast algorithm of the SCC particularly challenging. To efficiently reduce the computational complexity of SCC in VVC, we propose a deep learning based fast prediction network, namely FastSCCNet, where a fully convolutional network (FCN) is designed. CUs are classified into natural content block (NCB) and screen content block (SCB). With the use of FCN, only one shot inference is needed to classify the block types of the current CU and all corresponding sub-CUs. After block classification, different subsets of coding modes are assigned according to the block type, to accelerate the encoding process. Compared with the conventional SCC in VVC, our proposed FastSCCNet reduced the encoding time by 29.88% on average, with negligible bitrate increase under all-intra configuration. To the best of our knowledge, it is the first approach to tackle the computational complexity reduction for SCC in VVC.

I. Introduction

With the recent development of technologies in networking and portable devices, computer screen sharing applications have become much more popular. The applications includes remote desktop, video conferencing with document and slideshow sharing. These videos contain the mixture of camera-captured content (CC) and screen content (SC) as depicted in Fig. 1. In addition, there are also many TV programs and Internet videos which contain both CC and SC. There will be even cloud services using screen sharing technology in the coming future [1]. Efficient compression of SC is highly demanded. Thus, in January 2014, there was Call for Proposal (CfP) [2] of screen content coding (SCC) [3] as the extension of High Efficiency Video Coding (HEVC) [4] by the Joint Collaborative Team on Video Coding (JCT-VC). Two major coding modes, intra block copy (IBC) [5]–[7] and palette (PLT) [8]–[9] modes, were introduced to improve the coding efficiency. However, since the SCC extension was developed after HEVC released, SCC in HEVC is not widespread. Recently, the Joint Video Experts Team (JVET) started to develop the next-generation video coding standard, i.e. Versatile Video Coding (VVC) [10]–[11]. IBC has been included in VVC at the early stage while PLT has just been supported at recent development. The common test condition (CTC) for the compression of SC has also just been developed in April 2020 [13]. Wide range of SC applications using VVC with the enabling of IBC and PLT are expected in the future when VVC is finalized and released. However, the computational complexity of VVC increases by 18 times over that of HEVC under all intra test configuration [15] because of the introduction of new coding tools in VVC. One of the major tools is the new quad-tree plus multi-type tree (QTMT) coding structure for coding unit (CU) partitioning which largely increases the computational complexity. With IBC and PLT modes, the computational complexity is even higher. Hence, it is necessary to develop the fast approach for reducing the computational complexity of VVC in order to meet the needs of practical applications.