Multi-Scale Non-Local Sparse Attention for Single Image Super-Resolution

The non-local attention (NLA) has demonstrated its success in deep learning to solve various image processing and computer vision tasks. However, the NLA over the entire input image requires extremely high computational complexity and inevitably introduces irrelevant information since all the feature points are involved in calculating attention map. To address these problems, a novel attention module, called the multi-scale non-local sparse attention (MNSA), is proposed in this paper. In our MNSA, attention calculation is constrained within non-overlapping windows, and then only the most relevant feature points are selected to compute an attention map. The resulting sparse attention prevents the model from attending to irrelevant information and noise while reducing the computational complex-ity from quadratic to linear with respect to the input image size. To obtain receptive fields at different scales, our MNSA is further performed by exploiting different sizes of windows. Moreover, a novel local feature extraction (LFE) is proposed to extract the local structural information of natural images. To verify the effectiveness of our proposed attention module, a MNSA network is finally developed for conducting single image super-resolution (SISR). Extensive experimental results have clearly shown that our MNSA network can deliver superior performance over a number of state-of-the-art SISR methods.