Photo-Realistic Image Super-Resolution via Variational Autoencoders

There is a great leap in objective accuracy on image super-resolution, which recently brings a new challenge on image super-resolution with larger up-scaling (e.g. 4×) using pixel based distortion for measurement. This causes over-smooth effect which cannot grasp well the perceptual similarity. The advent of generative adversarial networks makes it possible super-resolve a low-resolution image to generate photo-realistic images sharing distribution with the high-resolution images. However, generative networks suffer from problems of mode-collapse and unrealistic sample generation. We propose to perform Image Super-Resolution via Variational AutoEncoders (SR-VAE) learning according to the conditional distribution of the high-resolution images induced by the low-resolution images. Given that the Conditional Variational Autoencoders tend to generate blur images, we add the conditional sampling mechanism to narrow down the latent subspace for reconstruction. To evaluate the model generalization, we use KL loss to measure the divergence between latent vectors and standard Gaussian distribution. Eventually, in order to balance the trade-off between super-resolution distortion and perception, not only that we use pixel based loss, we also use the modified deep feature loss between SR and HR images to estimate the reconstruction. In experiments, we evaluated a large number of datasets to make comparison with other state-of-the-art super-resolution approaches. Results on both objective and subjective measurements show that our proposed SR-VAE can achieve good photo-realistic perceptual quality closer to the natural image manifold while maintain low distortion.

I. Introduction

Image Super-Resolution (SR) is a fundamental problem in image processing. It is widely used in various image applications. With the advent of high-definition and ultra-definition images and videos, fast and accurate image super-resolution can ease the burden of broadcast and storage.