Spectral–Spatial Dual Graph Unfolding Network for Multispectral and Hyperspectral Image Fusion

Recently, deep neural network (DNN)-based methods have achieved good results in terms of the fusion of low-spatial-resolution hyperspectral (LR HS) and high-spatial-resolution multispectral (HR MS) images. However, the spectral band correlation (SBC) and the spatial nonlocal similarity (SNS) in hyperspectral (HS) images are not sufficiently exploited by them. To model the two priors efficiently, we propose a spectral–spatial dual graph unfolding network (SDGU-Net), which is derived from the optimization of graph regularized restoration models. Specifically, we introduce spectral and spatial graphs to regularize the reconstruction of the desired high-spatial-resolution HS (HR HS) image. To explore the SBC and SNS priors of HS images in feature space and utilize the powerful learning ability of DNNs simultaneously, the iterative optimization of the spectral and spatial graph regularized models is unfolded as a network, which is composed of spectral and spatial graph unfolding modules. The two kinds of modules are designed according to the solutions of the spectral and spatial graph regularized models. In these modules, we employ graph convolution networks (GCNs) to capture the SBC and SNS in the fused image. Then, the learned features are integrated by the corresponding feature fusion modules and fed into the feature condense module to generate the HR HS image. We conduct extensive experiments on three benchmark datasets, and the results demonstrate the effectiveness of our proposed SDGU-Net.

I. Introduction

Hyperspectral (HS) imaging technology can acquire tens or hundreds of spectral bands to describe the observed scenes, which has gradually become one of the most vital achievements in the field of remote sensing [1]. The obtained HS image not only records the spatial information of these scenes but also describes the spectral radiation information along the wavelength. Because of the abundant spectral information, the objects in the observed scenes are able to be identified by HS images, which have been applied to many fields, such as classification [2], detection [3], and mapping [4]. However, considering the physical limitations of remote sensing imaging systems, it is impractical to ensure that the HS images have both high spectral and spatial resolutions at the same time [5]. As an alternative solution, low-spatial-resolution HS (LR HS) images and high-spatial-resolution multispectral (HR MS) images are fused to generate high-spatial-resolution HS (HR HS) images [6], [7].