Data Acquisition and Preparation for Dual-Reference Deep Learning of Image Super-Resolution

The performance of deep learning based image super-resolution (SR) methods depend on how accurately the paired low and high resolution images for training characterize the sampling process of real cameras. Low and high resolution (LR $\sim $ HR) image pairs synthesized by degradation models (e.g., bicubic downsampling) deviate from those in reality; thus the synthetically-trained DCNN SR models work disappointingly when being applied to real-world images. To address this issue, we propose a novel data acquisition process to shoot a large set of LR $\sim $ HR image pairs using real cameras. The images are displayed on an ultra-high quality screen and captured at different resolutions. The resulting LR $\sim $ HR image pairs can be aligned at very high sub-pixel precision by a novel spatial-frequency dual-domain registration method, and hence they provide more appropriate training data for the learning task of super-resolution. Moreover, the captured HR image and the original digital image offer dual references to strengthen supervised learning. Experimental results show that training a super-resolution DCNN by our LR $\sim $ HR dataset achieves higher image quality than training it by other datasets in the literature. Moreover, the proposed screen-capturing data collection process can be automated; it can be carried out for any target camera with ease and low cost, offering a practical way of tailoring the training of a DCNN SR model separately to each of the given cameras.

I. Introduction

Single image super-resolution (SR), the task of increasing the spatial resolution and details of a given image, has attracted great attention from the computer vision research communities in the last few decades [1], [2]. In recent years, the state of the art of SR has been set and reset by various deep convolutional neural network (DCNN) based techniques [3]–[5]. In addition to the significant improvement in SR performance, these DCNN techniques also provide some important insights on the designs of DCNN architec- tures [6]–[9] and loss functions [4], [10]–[12]. However, for many real-world problems, the efficacy of a machine learning technique relies not only on the design of the technique itself but also, sometimes even more critically, on the quality and representativeness of the training data.