Fourier Space Losses for Efficient Perceptual Image Super-Resolution

Many super-resolution (SR) models are optimized for high performance only and therefore lack efficiency due to large model complexity. As large models are often not practical in real-world applications, we investigate and propose novel loss functions, to enable SR with high perceptual quality from much more efficient models. The representative power for a given low-complexity generator network can only be fully leveraged by strong guidance towards the optimal set of parameters. We show that it is possible to improve the performance of a recently introduced efficient generator architecture solely with the application of our proposed loss functions. In particular, we use a Fourier space supervision loss for improved restoration of missing high-frequency (HF) content from the ground truth image and design a discriminator architecture working directly in the Fourier domain to better match the target HF distribution. We show that our losses’ direct emphasis on the frequencies in Fourier-space significantly boosts the perceptual image quality, while at the same time retaining high restoration quality in comparison to previously proposed loss functions for this task. The performance is further improved by utilizing a combination of spatial and frequency domain losses, as both representations provide complementary information during training. On top of that, the trained generator achieves comparable results with and is 2.4× and 48× faster than state-of-the-art perceptual SR methods RankSRGAN and SRFlow respectively.

1. Introduction

Super-resolution (SR) deals with the problem of reconstructing the high-frequency (HF) information from a low-resolution (LR) image x ∈ ℝ^H×W×C, which are inherently lost after downsampling the high-resolution (HR) image y ∈ ℝ^rH×rW×C due to the lower Nyquist frequency in the LR space (r denotes the scaling factor). Recent single image SR (SISR) methods [4], [17], [22], [19], [10], [14] have shown remarkable success at reconstructing the missing HF details, with emphasis on accurate restoration of the frequency content in the ground truth frames. This is typically performed with supervised training, where the ground truth images y are downsampled with a known kernel, e.g. bicubic, to obtain the LR input images x.