Spatially-Adaptive Feature Modulation for Efficient Image Super-Resolution

Although deep learning-based solutions have achieved impressive reconstruction performance in image super-resolution (SR), these models are generally large, with complex architectures, making them incompatible with low-power devices with many computational and memory constraints. To overcome these challenges, we propose a spatially-adaptive feature modulation (SAFM) mechanism for efficient SR design. In detail, the SAFM layer uses independent computations to learn multi-scale feature representations and aggregates these features for dynamic spatial modulation. As the SAFM prioritizes exploiting non-local feature dependencies, we further introduce a convolutional channel mixer (CCM) to encode local contextual information and mix channels simultaneously. Extensive experimental results show that the proposed method is 3× smaller than state-of-the-art efficient SR methods, e.g., IMDN, and yields comparable performance with much less memory usage. Our source codes and pre-trained models are available at: https://github.com/sunny2109/SAFMN.

1. Introduction

Single image super-resolution (SISR) aims to restore a high-resolution (HR) image from its low-resolution (LR) counterpart by recovering lost details. This longstanding and challenging task has recently attracted much attention due to the rapid development of streaming media or high-definition devices. As these scenarios are usually resource-constrained, it is of great interest to develop an efficient and effective SR method to estimate HR images for better visual display on these platforms or products.