In-Loop Filtering via Trained Look-Up Tables

In-loop filtering (ILF) is a key technology in image/video coding for reducing the artifacts. Recently, neural network-based in-loop filtering methods achieve remarkable coding gains beyond the capability of advanced video coding standards, establishing themselves a promising candidate tool for future standards. However, the utilization of deep neural networks (DNN) brings high computational complexity and raises high demand of dedicated hardware, which is challenging to apply into general use. To address this limitation, we study an efficient in-loop filtering scheme by adopting look-up tables (LUTs). After training a DNN with a predefined reference range for in-loop filtering, we cache the output values of the DNN into a LUT via traversing all possible inputs. In the coding process, the filtered pixel is generated by locating the input pixels (to-be-filtered pixel and reference pixels) and interpolating between the cached values. To further enable larger reference range within the limited LUT storage, we introduce an enhanced indexing mechanism in the filtering process, and a clipping/finetuning mechanism in the training. The proposed method is implemented into the Versatile Video Coding (VVC) reference software, VTM-11.0. Experimental results show that the proposed method, with three different configurations, achieves on average 0.13%∼0.51%, and 0.10% ∼0.39% BD-rate reduction under the all-intra (AI) and random-access (RA) configurations respectively. The proposed method incurs only 1% ∼8% time increase, an additional computation of 0.13 ∼0.93 kMAC/pixel, and 164 ∼1148 KB storage cost for a single model. Our method has explored a new and more practical approach for neural network-based ILF.

I. Introduction

In-loop filtering (ILF) has been widely adopted in modern video coding standards, including H.266/VVC [1], AV2 [2]. To promote the reconstruction quality of decoded frame, various complementary filters make a major contribution to these standards and play a key role in hybrid video coding framework, such as deblocking filter (DBF), sample adaptive offset (SAO), adaptive loop filtering (ALF) [3].