Multi–Grid Back–Projection Networks

Multi–grid back–projection (MGBP) is a fully–convolutional network architecture that can learn to restore images and videos with upscaling artifacts. Using the same strategy of multi–grid partial differential equation (PDE) solvers this multiscale architecture scales computational complexity efficiently with increasing output resolutions. The basic processing block is inspired in the iterative back–projection (IBP) algorithm and constitutes a type of cross–scale residual block with feedback from low resolution references. The architecture performs in par with state–of–the-arts alternatives for regression targets that aim to recover an exact copy of a high resolution image or video from which only a downscale image is known. A perceptual quality target aims to create more realistic outputs by introducing artificial changes that can be different from a high resolution original content as long as they are consistent with the low resolution input. For this target we propose a strategy using noise inputs in different resolution scales to control the amount of artificial details generated in the output. The noise input controls the amount of innovation that the network uses to create artificial realistic details. The effectiveness of this strategy is shown in benchmarks and it is explained as a particular strategy to traverse the perception–distortion plane.

I. Introduction

Image and video upscaling has been studied for decades and remains an active topic of research because of constant technological advances in digital imaging. One scenario where upscaling is now more demanding arises in digital display technologies, where new standards like BT.2020 [1] are introduced. The resolution of digital displays has experienced a tremendous growth over the past few decades, as shown in Fig. 1. The transition between different formats leads to a challenging problem. On one hand, large amount of digital content still exist in popular old standards such as standard–definition (SD). On the other hand, the latest display technologies (e.g. 4 K, 8 K and above) are expected to show this content with reasonable quality. Standard upscaling technologies are clearly insufficient for this purpose. While a upscaler maps 1 input pixel into 4 output pixels, an upscaler maps 1 input pixel into 64 output pixels, which already can contain a small image. The problem demands advanced solutions that are capable of understanding the content and filling in these large pieces of images with visually appealing and consistent information. In particular, large upscaling factors are needed to convert SD to ultra high–definition (UHD) resolutions. Thus, large upscaling represents a real problem in current market and it is expected to persist and become even more challenging with the rapid development of new technologies.