Unsupervised Adaptation Learning for Hyperspectral Imagery Super-Resolution

The key for fusion based hyperspectral image (HSI) super-resolution (SR) is to infer the posteriori of a latent HSI using appropriate image prior and likelihood that depends on degeneration. However, in practice the priors of high-dimensional HSIs can be extremely complicated and the degeneration is often unknown. Consequently most existing approaches that assume a shallow hand-crafted image prior and a pre-defined degeneration, fail to well generalize in real applications. To tackle this problem, we present an unsupervised adaptation learning (UAL) framework. Instead of directly modelling the complicated image prior, we propose to first implicitly learn a general image prior using deep networks and then adapt it to a specific HSI. Following this idea, we develop a two-stage SR network that leverages two consecutive modules: a fusion module and an adaptation module, to recover the latent HSI in a coarse-to-fine scheme. The fusion module is pretrained in a supervised manner on synthetic data to capture a spatial-spectral prior that is general across most HSIs. To adapt the learned general prior to the specific HSI under unknown degeneration, we introduce a simple degeneration network to assist learning both the adaptation module and the degeneration in an unsupervised way. In this way, the resultant image-specific prior and the estimated degeneration can benefit the inference of a more accurate posteriori, thereby increasing generalization capacity. To verify the efficacy of UAL, we extensively evaluate it on four benchmark datasets and report strong results that surpass existing approaches.

1. Introduction

Hyperspectral images (HSIs) consists of hundreds of spectral bands that record the reflectance of an imaging scene across a consecutive wavelengths with narrow interval (e.g. 10nm) [1], [32], where each pixel contains a spectrum. Due to the discriminative power of their spectra [13], HSIs have been widely applied to various computer vision tasks, such as target detection [17], classification [13], tracking [22] etc. However, physical limitations on spectral sensor [12], [7] often prevent the collection of high resolution (HR) HSIs in practice. Hence increasing efforts have been put forwards to HSI super-resolution (SR).