NTIRE 2021 Challenge on Perceptual Image Quality Assessment

This paper reports on the NTIRE 2021 challenge on perceptual image quality assessment (IQA), held in conjunction with the New Trends in Image Restoration and Enhancement workshop (NTIRE) workshop at CVPR 2021. As a new type of image processing technology, perceptual image processing algorithms based on Generative Adversarial Networks (GAN) have produced images with more realistic textures. These output images have completely different characteristics from traditional distortions, thus pose a new challenge for IQA methods to evaluate their visual quality. In comparison with previous IQA challenges, the training and testing datasets in this challenge include the outputs of perceptual image processing algorithms and the corresponding subjective scores. Thus they can be used to develop and evaluate IQA methods on GAN-based distortions. The challenge has 270 registered participants in total. In the final testing stage, 13 participating teams submitted their models and fact sheets. Almost all of them have achieved much better results than existing IQA methods, while the winning method can demonstrate state-of-the-art performance.

1. Introduction

Image quality assessment (IQA) aims at using computational models to measure the perceptual quality of images, which are degraded during acquisition, compression, reproduction and post-processing operations. As the "evaluation mechanism", IQA plays a critical role in most image processing tasks, such as image super-resolution, denoising, compression and enhancement. Although it is easy for human beings to distinguish perceptually better images, it has been proved to be difficult for algorithms [38], [20]. Especially, on the basis of Generative Adversarial Networks (GANs) [18], perceptual image processing algorithms (or perceptual-oriented algorithms) [25], [30], [53], [62] have posed a great challenge for IQA methods, as they bring completely new characteristics to the output images [20]. It has been noticed that the contradiction between the quantitative evaluation results and the real perceptual quality is increasing [6], [7], [20]. This will also affect the development of image processing algorithms, if the IQA methods cannot objectively compare their perceptual quality [7], [20]. Therefore, new IQA methods need to be proposed accordingly, to adapt new image processing algorithms.