MFT-GAN: A Multiscale Feature-Guided Transformer Network for Unsupervised Hyperspectral Pansharpening

Unsupervised learning, learning data distributions without needing labeled samples, is a particularly promising approach for solving the challenging task of hyperspectral pansharpening. Inspired by the above, we introduce an innovative generative adversarial network framework (named MFT-GAN) that incorporates a transformer network and multiscale interaction technology. Specifically, MFT-GAN is constructed of one generator and two discriminators. The generator is composed of a multiscale feature guidance branch (MFGB) and a feature interaction fusion branch (FIFB). The former aims at extracting spectral and spatial information branches from low-resolution hyperspectral (LRHS) and panchromatic (PAN) images by convolutional sampling operations, respectively. The latter accomplishes the interaction of spectral and spatial information at different scales through multiscale interaction technology. Then, the multiscale interaction results are fused by level-by-level convolutional sampling and pixel summation operations to generate high-resolution hyperspectral (HRHS) images. The discriminators are composed of a spectral transformer discriminator and a spatial transformer discriminator, designed to maintain structural and parametric balance and learn long-range spectral and spatial correlation relationships. In addition, hybrid loss functions are used to complete the adversarial training of the MFT-GAN to improve its performance. The experiments on simulated and real datasets further demonstrated the effectiveness of the proposed MFT-GAN method.

I. Introduction

Hyperspectral pansharpening [1] is a technique that produces high-resolution hyperspectral (HRHS) images by integrating low-resolution hyperspectral (LRHS) images with panchromatic (PAN) images. Remote sensing imaging systems can provide only high spatial resolution images (e.g., PAN images) or high spectral resolution images (e.g., LRHS images), driving the development of sharpening techniques. Moreover, the information contained in images can be improved by combining hyperspectral information from LRHS images with high-resolution PAN information from PAN images, which helps analyze and identify target features more accurately. Pansharpening has contributed to many fields, such as classification [2], [3], [4], unmixing [5], [6], [7], [8], and detection [9], [10], [11].