RankSRGAN: Super Resolution Generative Adversarial Networks With Learning to Rank

Generative Adversarial Networks (GAN) have demonstrated the potential to recover realistic details for single image super-resolution (SISR). To further improve the visual quality of super-resolved results, PIRM2018-SR Challenge employed perceptual metrics to assess the perceptual quality, such as PI, NIQE, and Ma. However, existing methods cannot directly optimize these indifferentiable perceptual metrics, which are shown to be highly correlated with human ratings. To address the problem, we propose Super-Resolution Generative Adversarial Networks with Ranker (RankSRGAN) to optimize generator in the direction of different perceptual metrics. Specifically, we first train a Ranker which can learn the behaviour of perceptual metrics and then introduce a novel rank-content loss to optimize the perceptual quality. The most appealing part is that the proposed method can combine the strengths of different SR methods to generate better results. Furthermore, we extend our method to multiple Rankers to provide multi-dimension constraints for the generator. Extensive experiments show that RankSRGAN achieves visually pleasing results and reaches state-of-the-art performance in perceptual metrics and quality. Project page: https://wenlongzhang0517.github.io/Projects/RankSRGAN.

1 Introduction

Single image super resolution aims at reconstructing/generating a high-resolution (HR) image from a low-resolution (LR) observation. Thanks to the strong learning capability, Convolutional Neural Networks (CNNs) have demonstrated superior performance [1], [2], [3], [4], [5], [6], [7], [8] to the conventional example-based [9], [10], [11], [12], [13] and interpolation-based [14], [15], [16] algorithms. Recent CNN-based methods can be divided into two groups. The first one regards SR as a reconstruction problem and adopts mean squared error (MSE) as the loss function to achieve high PSNR values. However, due to the conflict between the reconstruction accuracy and visual quality, they tend to produce overly smoothed/sharpened images. To favour better visual quality, the second group casts SR as an image generation problem [17]. By incorporating the perceptual loss [18], [19] and adversarial learning [17], these perceptual SR methods have the potential to generate realistic textures and details, thus attracted increasing attention in recent years.