Unsupervised Test-Time Adaptation Learning for Effective Hyperspectral Image Super-Resolution With Unknown Degeneration

Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) multi-spectral image has provided an effective way for HSI super-resolution (SR). The key lies on inferring the posteriori of the latent (i.e., HR) HSI using an appropriate image prior and the likelihood determined by the degeneration between the latent HSI and the observed images. However, in scenarios with complex imaging environments and various imaging scenes, the prior of HSIs can be prohibitively complicated and the degeneration is often unknown, which causes it difficult to accurately infer the posteriori of each latent HSI. To tackle this problem, we present an unsupervised test-time adaptation learning (UTAL) framework for HSI SR under unknown degeneration. Instead of directly modeling the complicated image prior, it first implicitly learns a content-agnostic prior shared across different images through supervisedly pre-training a mutual-guiding fusion module on extensive synthetic data. Then, it adapts the shared prior to those private characteristics in the latent HSI for posteriori inference through unsupervisedly learning a self-guiding adaptation module and a degeneration estimation network on two observed images in the test phase. Such a two-stage learning scheme models the complicated image prior in a divide-and-conquer manner, which eases the modeling difficulty and improves the prior accuracy. Moreover, the unknown degeneration can be estimated properly. Both of these two advantages empower us to accurately infer the posteriori of the latent HSI, thereby increasing the generalization performance in real applications. Additionally, in order to further mitigate the over-fitting in coping with more challenging cases (e.g., degenerations in both spectral and spatial domains are unknown) and speed up, we propose to meta-train UTAL on extensive synthetic SR tasks and solve it using an alternative optimization strategy such that UTAL learns to produce good generalization performance in real challenging cases with a small number of gradient descent steps. To verify the efficacy of UTAL, we evaluate it on HSI SR tasks with different unknown degenerations as well as some other HSI restoration tasks (e.g., compressive sensing), and report strong results superior to that of existing competitors.

I. Introduction

Hyperspectral images (HSIs) record the reflectance of imaging scenes across a consecutive wavelength with narrow interval (e.g.10 nm) using hundreds of spectral bands [1], [2], and each pixel contains a spectrum that can finely depict the physical reflectance characteristics of the specific position in scenes. Profiting from such unique discriminative power of spectra [3], HSIs have been widely utilized in both natural scenes [4], [5], [6], [7] and remote sensing scenes [8], [9], [10], [11], [12], [13], [14], and shown great potential in plenty of applications like object tracking [15], [16], [17], food safety protection [18], [19], space exploration [20], military reconnaissance [21], ecological protection [19], [22], and precision agriculture [23], etc.. However, due to the physical limitations on spectral sensor [24], [25], only HSIs of limited resolution (e.g., multiple or even tens of times lower than RGB images [26], [27]) can be captured in practice, which incurs obvious performance degradation in HSIs related tasks. Hence increasing efforts have been put into HSI super-resolution (SR) which aims to increase the resolution of HSIs in a post-processing manner without any change on the physical sensors.