Real-Time Quantized Image Super-Resolution on Mobile NPUs, Mobile AI 2021 Challenge: Report

Image super-resolution is one of the most popular computer vision problems with many important applications to mobile devices. While many solutions have been proposed for this task, they are usually not optimized even for common smartphone AI hardware, not to mention more constrained smart TV platforms that are often supporting INT8 inference only. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based image super-resolution solutions that can demonstrate a real-time performance on mobile or edge NPUs. For this, the participants were provided with the DIV2K dataset and trained quantized models to do an efficient 3X image upscaling. The runtime of all models was evaluated on the Synaptics VS680 Smart Home board with a dedicated NPU capable of accelerating quantized neural networks. The proposed solutions are fully compatible with all major mobile AI accelerators and are capable of reconstructing Full HD images under 40-60 ms while achieving high fidelity results. A detailed description of all models developed in the challenge is provided in this paper.

1. Introduction

Image super-resolution is a classical computer vision problem where the goal is to reconstruct the original image based on its downscaled version, adding the lost lost high frequencies and rich texture details. During the past years, this task has witnessed an increased popularity due to its direct application to telephoto image processing in smartphone cameras, low-resolution media data enhancement as well as to upscaling images and videos to the target high resolution of display panels. Numerous classical [36], [14], [46], [50], [51], [57], [59], [60], [16], [53] and deep learning-based [12], [11], [39], [41], [49], [52], [6], [44], [61], [31] approaches have been proposed for this task in the past. The biggest limitation of these methods is that they were primarily targeted at achieving high fidelity scores while not optimized for computational efficiency and mobile-related constraints, which is essential for this and other tasks related to image processing and enhancement [19], [20], [33] on mobile devices. In this challenge, we take one step further in solving this problem by using a popular DIV2K [3] image super-resolution dataset and by imposing additional efficiency-related constraints on the developed solutions.