Model-Guided Deep Hyperspectral Image Super-Resolution

The trade-off between spatial and spectral resolution is one of the fundamental issues in hyperspectral images (HSI). Given the challenges of directly acquiring high-resolution hyperspectral images (HR-HSI), a compromised solution is to fuse a pair of images: one has high-resolution (HR) in the spatial domain but low-resolution (LR) in spectral-domain and the other vice versa. Model-based image fusion methods including pan-sharpening aim at reconstructing HR-HSI by solving manually designed objective functions. However, such hand-crafted prior often leads to inevitable performance degradation due to a lack of end-to-end optimization. Although several deep learning-based methods have been proposed for hyperspectral pan-sharpening, HR-HSI related domain knowledge has not been fully exploited, leaving room for further improvement. In this paper, we propose an iterative Hyperspectral Image Super-Resolution (HSISR) algorithm based on a deep HSI denoiser to leverage both domain knowledge likelihood and deep image prior. By taking the observation matrix of HSI into account during the end-to-end optimization, we show how to unfold an iterative HSISR algorithm into a novel model-guided deep convolutional network (MoG-DCN). The representation of the observation matrix by subnetworks also allows the unfolded deep HSISR network to work with different HSI situations, which enhances the flexibility of MoG-DCN. Extensive experimental results are reported to demonstrate that the proposed MoG-DCN outperforms several leading HSISR methods in terms of both implementation cost and visual quality. The code is available at https://see.xidian.edu.cn/faculty/wsdong/Projects/MoG-DCN.htm.

I. Introduction

Hyperspectral image (HSI) containing a large number of spectral bands has advantages over commonly used multispectral images (MSI) such as RGB data when identifying the physical properties of object materials [1]. Accordingly, HSI has found many successful applications in computer vision from image segmentation [2] and object recognition [3] to image classification [4] and visual tracking [5]–[7]. The trade-off between spatial and spectral resolution has remained one of the great challenges in the practice of HS imaging. Due to various hardware and budget constraints, it is difficult to directly acquire images that have high-resolution (HR) in both spatial and spectral domains. Accordingly, computational approaches to improve the quality of low-resolution (LR) images for HS imaging, such as super-resolution (SR) and pan-sharpening [8], [9] have attracted increasingly more attention. For example, the class of HS Imaging Super-Resolution (HSISR) methods aim at obtaining a HR-HS image by fusing a LR-HSI with a HR-MSI [10].