Residual Feature Aggregation Network for Image Super-Resolution

Recently, very deep convolutional neural networks (CNNs) have shown great power in single image super-resolution (SISR) and achieved significant improvements against traditional methods. Among these CNN-based methods, the residual connections play a critical role in boosting the network performance. As the network depth grows, the residual features gradually focused on different aspects of the input image, which is very useful for reconstructing the spatial details. However, existing methods neglect to fully utilize the hierarchical features on the residual branches. To address this issue, we propose a novel residual feature aggregation (RFA) framework for more efficient feature extraction. The RFA framework groups several residual modules together and directly forwards the features on each local residual branch by adding skip connections. Therefore, the RFA framework is capable of aggregating these informative residual features to produce more representative features. To maximize the power of the RFA framework, we further propose an enhanced spatial attention (ESA) block to make the residual features to be more focused on critical spatial contents. The ESA block is designed to be lightweight and efficient. Our final RFANet is constructed by applying the proposed RFA framework with the ESA blocks. Comprehensive experiments demonstrate the necessity of our RFA framework and the superiority of our RFANet over state-of-the-art SISR methods.

1. Introduction

The task of single image super-resolution (SISR) is to map a degraded low-resolution (LR) image to a visually high-resolution (HR) image, which is a highly ill-posed procedure since multiple HR solutions can map to one LR input. Many image SR methods have been proposed to tackle this inverse problem, including early interpolation-based [37], reconstruction-based [34], and recent learning based methods [27], [28], [22], [4], [12], [13], [36], [3]. Figure 1.

(a) A chain of residual modules. A residual module consists of a residual block (RB) and an identity connection. (b) The residual feature aggregation (RFA) framework.