Multi-Level Encoder-Decoder Architectures for Image Restoration

Many real-world solutions for image restoration are learning-free and based on handcrafted image priors such as self-similarity. Recently, deep-learning methods that use training data have achieved state-of-the-art results in various image restoration tasks (e.g., super-resolution and inpainting). Ulyanov et al. bridge the gap between these two families of methods (CVPR 18). They have shown that learning-free methods perform close to the state-of-the-art learning-based methods (approximately 1 PSNR). Their approach benefits from the encoder-decoder network. In this paper, we propose a framework based on the multi-level extensions of the encoder-decoder network, to investigate interesting aspects of the relationship between image restoration and network construction independent of learning. Our framework allows various network structures by modifying the following network components: skip links, cascading of the network input into intermediate layers, a composition of the encoder-decoder subnetworks, and network depth. These handcrafted network structures illustrate how the construction of untrained networks influence the following image restoration tasks: denoising, super-resolution, and inpainting. We also demonstrate image reconstruction using flash and no-flash image pairs. We provide performance comparisons with the state-of-the-art methods for all the restoration tasks above.

1. Introduction

Image restoration is an ill-posed problem which aims to recover an image given its corrupted observation (e.g., denoising [39], [5], [31], super-resolution [14], [23], [2], and inpainting [36], [34], [33]). Corruption may occur due to noise, camera shake, and due to the fact that the picture was taken in rain or underwater [16]. Image restoration methods could be mainly classified into two types - traditional methods and deep-learning (DL) methods. Traditional methods include spatial filtering methods (e.g., bilateral filters [28], non-local means [4]), wavelet transform based methods [6], and dictionary learning and sparse coding [17], [37]. DL methods generally include a neural network to learn image prior from the training samples (learning-based¹

The learning refers to training the network on the collection of images and learning-free refers to the methods which do not use training data.

) for restoration, where the training samples contain paired examples of corrupted and high-quality images.