SS-INR: Spatial-Spectral Implicit Neural Representation Network for Hyperspectral and Multispectral Image Fusion

Due to the limitation of imaging equipment, it is difficult to acquire hyperspectral images (HSIs) with high spatial resolution directly. Existing approaches improve the resolution of HSIs by fusing multispectral image (MSI) and HSI. However, most of them are only feed-forward. They only learn low-resolution (LR) to high-resolution (HR) feature mappings without considering the ill-posedness of super-resolution tasks, leading to a large solution space of mapping functions and making it difficult to learn a complete mapping function. Moreover, there is a large resolution difference between HSI and MSI, and some up-sampling operations are inevitably employed in the network. Nevertheless, traditional upsampling methods only represent pixel points in a discrete way, failing to adequately restore the continuous spatial and spectral information. To this end, this article proposes a spatial-spectral implicit neural representation network for hyperspectral and multispectral image fusion (SS-INR). Inspired by the success of implicit neural representation (INR) in continuum reconstruction, we design spatial-INR and spectral-INR for spatial and spectral resolution reconstruction, respectively. SS-INR contains two processes: forward fusion (FF) and back-projection fusion (BPF). In the FF process, the input HSI is first spatially upsampled with spatial-INR to overcome spatial resolution differences while performing initial fusion with MSI. In the BPF process, we explore the spatial and spectral degradation processes and use them as prior knowledge for error correction. Extensive experiments on five public hyperspectral datasets demonstrate the effectiveness of SS-INR, and SS-INR achieves competitive results compared with existing state-of-the-art (SOTA) fusion methods. The source code for SS-INR will be released at https://github.com/wxy11-27/SS-INR.

I. Introduction

In remote sensing imaging, a larger instantaneous field-of-view (IFOV) must be utilized to achieve acceptable signal-to-noise ratio when dealing with narrower spectral bandwidths. This implies that while maintaining the spectral resolution of the imaging, the spatial resolution is inevitably reduced. However, in practical applications of high-precision remote sensing such as target detection [1], change detection [2], [3], and land cover classification [4], [5], both spatial resolution and spectral resolution hold importance. However, we cannot directly obtain HSIs with high spatial resolution due to the incompatibility of spatial and spectral resolution. Under this circumstance, the HSI and MSI fusion technology has arisen as a means of generating HSI with high spatial resolution. It is achieved by integrating the spatial information from MSI with the spectral information inherent to HSI.