Transformer-based Deep Frame Rate up Conversion for Hybrid High-Efficiency Video Coding

Mainstream state-of-the-art video compression standards such as High-Efficiency Video Coding (HEVC/H.265) and Versatile Video Coding (VVC/H.266) are block-based coding. On the other hand, neural video compression networks have obtained impressive results recently. However, neural video compression is not practical for use due to its high computations. In this paper, we propose a hybrid approach that exploits a deep frame rate up-conversion network to improve the coding efficiency of existing video codecs such as HEVC. We analyze a coding scenario of a combination between the Frame Rate Up-Conversion (FRUC) network and the random access coding configuration. In addition, we show that the FRUC network trained on compressed images outperforms the model trained on non-compressed images for the video frame interpolation task. The proposed framework skips encoding odd frames in the display order of original video sequences at the encoder side and reconstructs them at the decoder side via the FRUC network. Experimental results show the proposed framework improves the coding efficiency of HEVC 4.2 % for random access configuration.