ConvGRU-Based Multiscale Frequency Fusion Network for PAN-MS Joint Classification

As a hot research topic in remote sensing, effectively integrating the advantageous features of multispectral and panchromatic images is the main challenge for fusing these two remote sensing images. This article proposes a multiscale frequency fusion network based on ConvGRU. To address the underutilization of texture features, we extract multiscale bandpass and low-pass sub-bands representing texture and content features through Contourlet decomposition. Multiscale bandpass sub-bands contain more comprehensive and concentrated texture details. Then, by proposing a multiscale frequency feature extractor based on ConvGRU, we effectively integrate and enhance sub-bands of different scales and frequencies, fully utilizing the characteristics of multispectral and panchromatic images and scale transmission. With these enhanced sub-band features, we obtain more comprehensive scale-enhanced texture features. Simultaneously, content features are also preserved as dual-source image features. Moreover, to reduce redundancy between fused features and make more efficient use of the obtained enhanced features, we designed an Inver-band integrator (IBI) module. It can fuse enhanced features at different scales, improve the complementarity between features, and thus achieve effective fusion. Experimental results demonstrate the effectiveness and robustness of our model on multiple datasets. Our codes are available at https://github.com/Xidian-AIGroup190726/GMFnet.

I. Introduction

In recent years, with the continuous development of remote sensing satellite technology [1], [2], the application scenarios of remote sensing images have gradually expanded. People have also paid more attention to the properties of remote sensing images. Typical remote sensing satellites include Landsat 7 ETM+, Landsat 8 OLI, IKONOS, QuickBird, GaoFen-2, WorldView-3, etc. Due to physical limitations [3], for the same scene, remote sensing satellites can obtain multiple remote sensing images [4] with approximately similar content but completely different properties through remote sensing cameras [5]. These include multispectral images and panchromatic images [6]. Multispectral images contain rich spectral information, while panchromatic images have high spatial resolution. The hot research topic is how to fuse these two types of remote sensing images effectively.