Effect of Training and Test Datasets on Image Restoration and Super-Resolution by Deep Learning

Many papers have recently been published on image restoration and single-image super-resolution (SISR) using different deep neural network architectures, training methodology, and datasets. The standard approach for performance evaluation in these papers is to provide a single “average” mean-square error (MSE) and/or structural similarity index (SSIM) value over a test dataset. Since deep learning is data-driven, performance of the proposed methods depends on the size of the training and test sets as well as the variety and complexity of images in them. Furthermore, the performance varies across different images within the same test set. Hence, comparison of different architectures and training methods using a single average performance measure is difficult, especially when they are not using the same training and test sets. We propose new measures to characterize the variety and complexity of images in the training and test sets, and show that our proposed dataset complexity measures correlate well with the mean PSNR and SSIM values obtained on different test data sets. Hence, better characterization of performance of different methods is possible if the mean and variance of the MSE or SSIM over the test set as well as the size, resolution and complexity measures of the training and test sets are specified.

I. Introduction

Various linear, adaptive, and nonlinear filters have been developed for image restoration and single-image super-resolution (SISR) over the years [1]. It is well-known that linear filters are limited by their ability to trade-off noise amplification with regularization artifacts [2]. Adaptive restoration filters can control the amount of ringing artifacts by avoiding filtering across sharp edges (high spatial frequencies). Methods to avoid ringing originating from model-misfit at image boundaries were also discussed. Traditionally, different classical image restoration and SISR methods have been tested on a few standard images, such as Cameraman and Lena, and the mean square error (MSE) or peak-signal-to-noise ratio (PSNR) scores have been reported for evaluation and comparison of methods.