STCP: Synergistic Transformer and Convolutional Neural Network for Pansharpening

Pansharpening is a process of fusing a high-resolution panchromatic (PAN) image with a low-resolution multispectral (LRMS) image to obtain a high-resolution multispectral (HRMS) image. Convolutional neural networks (CNNs) have been commonly utilized in this field because of their remarkable learning capabilities. However, their convolutional operators limit the long-range feature extraction ability of CNN. Meanwhile, the transformer models have exhibited strong capabilities in modeling long-range representations, but there are shortcomings in modeling local-range feature dependencies. To this end, we propose a novel synergistic transformer and CNN for pansharpening (STCP). First, a parallel U-shaped feature extraction module (PUFEM) is constructed for extracting the features of the LRMS and PAN images, which improves the feature representation ability for the two source images. In the PUFEM, combining the different feature learning capabilities of the CNN and transformer, we design a long- and short-range feature integration block (LSFIB) to extract the short-range features and long-range features at different scales in parallel. Then, a channel attention module (CAM)-based feature fusion module (CFFM) is constructed to integrate the features extracted by the PUFEM. Finally, the shallow features from the PAN image are reused to provide detailed features, which are integrated with the fused features from the CFFM to achieve the final pansharpened results. Numerous experiments show that our STCP outperforms some state-of-the-art (SOTA) approaches both subjectively and objectively.

I. Introduction

Satellite images have been popularly applied in many fields, including Earth surface change detection [1], target recognition [2], and ground object classification [3]. Due to the limitations of imaging systems, however, it is impossible to obtain satellite images with both high spatial and spectral resolutions [4]. Thus, satellites usually utilize different sensors to provide low-resolution multispectral (LRMS) images and high-resolution panchromatic (PAN) images. Therefore, pansharpening technologies are required for fusing PAN and multispectral (MS) images to achieve high-resolution multispectral (HRMS) images.