Toward DNN of LUTs: Learning Efficient Image Restoration With Multiple Look-Up Tables

The widespread usage of high-definition screens on edge devices stimulates a strong demand for efficient image restoration algorithms. The way of caching deep learning models in a look-up table (LUT) is recently introduced to respond to this demand. However, the size of a single LUT grows exponentially with the increase of its indexing capacity, which restricts its receptive field and thus the performance. To overcome this intrinsic limitation of the single-LUT solution, we propose a universal method to construct multiple LUTs like a neural network, termed MuLUT. First, we devise novel complementary indexing patterns, as well as a general implementation for arbitrary patterns, to construct multiple LUTs in parallel. Second, we propose a re-indexing mechanism to enable hierarchical indexing between cascaded LUTs. Finally, we introduce channel indexing to allow cross-channel interaction, enabling LUTs to process color channels jointly. In these principled ways, the total size of MuLUT is linear to its indexing capacity, yielding a practical solution to obtain superior performance with the enlarged receptive field. We examine the advantage of MuLUT on various image restoration tasks, including super-resolution, demosaicing, denoising, and deblocking. MuLUT achieves a significant improvement over the single-LUT solution, e.g., up to 1.1 dB PSNR for super-resolution and up to 2.8 dB PSNR for grayscale denoising, while preserving its efficiency, which is 100× less in energy cost compared with lightweight deep neural networks.

I. Introduction

Image restoration aims to generate high-quality (HQ) visual data with high-frequency details from low-quality (LQ) observations (e.g., downscaled, noisy, and compressed images). Image restoration algorithms enjoy wide applications, ranging from visual quality enhancement [1], [2], digital holography [3], satellite imaging [4], medical imaging [5], and gaming [6]. Moreover, besides improving image quality, image restoration helps in many other computer vision tasks, e.g., human face recognition [7] and autonomous driving [8].