Reciprocal Attention Mixing Transformer for Lightweight Image Restoration

Although many recent works have made advancements in the image restoration (IR) field, they often suffer from an excessive number of parameters. Another issue is that most Transformer-based IR methods focus only on either local or global features, leading to limited receptive fields or deficient parameter issues. To address these problems, we propose a lightweight network, Reciprocal Attention Mixing Transformer (RAMiT). It employs our proposed dimensional reciprocal attention mixing Transformer (D-RAMiT) blocks, which compute bi-dimensional self-attentions in parallel with different numbers of multi-heads. The bi-dimensional attentions help each other to complement their counterpart’s drawbacks and are then mixed. Additionally, we introduce a hierarchical reciprocal attention mixing (H-RAMi) layer that compensating for pixel-level information losses and utilizes semantic information while maintaining an efficient hierarchical structure. Furthermore, we revisit and modify MobileNet V2 to attach efficient convolutions to our proposed components. The experimental results demonstrate that RAMiT achieves state-of-the-art performance on multiple lightweight IR tasks, including super-resolution, low-light enhancement, deraining, color denoising, and grayscale denoising. Codes are available at https://github.com/rami0205/RAMiT.

1. Introduction

Lightweight image restoration (IR) or enhancement techniques are essential for addressing inherent flaws in images captured in the wild, especially those taken by devices with low computational power. These techniques aim to reconstruct high-quality images from their distorted low-quality counterparts. However, many lightweight IR tasks with the popular vision Transformer [14] based methods remain relatively unexplored. Although many recent Transformer [52] networks have improved the IR domain [7], [9], [56], [65], [69], they are infeasible for real-world applications due to their large number of parameters. Furthermore, even the state-of-the-art lightweight IR networks consume intensive computational costs [5], [10], [33], [38], [73]. Another problem is that some IR models mainly focus on expanding the receptive field with respect to locality [9], [10], [33], [56], [73], which is insufficient to capture the global dependency in an image. This is critical because the IR networks need to refer to repeated patterns and textures distributed throughout the image [18], [38]. Meanwhile, others have tried to enlarge the receptive field globally [5], [65], [69] but have overlooked important local (spatial) information, which is conventionally essential for recovery tasks [9], [10], [21], [56]. Fig. 1 visualizes a few examples in which a successful IR depends on the ability to consider both local and global features in a given distorted low-quality image, emphasizing how significant the problem is.