High-Fidelity Hyperspectral Snapshot of Physical World: System Architecture, Dataset and Model

It is extremely challenging to acquire high-fidelity video of physical world at high spectral, high spatial and high temporal resolution (H$^{3}$R) granularities simultaneously. Existing hyperspectral scanning cameras offer groundtruth with sufficient spatiospectral resolutions but largely lack temporal details; while recent hyperspectral snapshot cameras (e.g., CASSI, PMVIS) enable high temporal resolution acquisition but present inferior capacity to measure fine-grained spatiospectral components as groundtruth. This work builds a joint snapshot-scanning spectral system (JS$^{4}$), by means of which the acquired low spatiospectral video-rate snapshots can be guided by synchronous-captured high spatiospectral groundtruth in the hyperspectral propagation. We first register the snapshot-scanning image pairs of the JS$^{4}$ using a physical imaging model, and then generate a computational hyperspectral image light dataset (CHILD) having both video-rate snapshot and corresponding scanned groundtruth of a specific scene. Taking advantage of the CHILD, we later develop an end-to-end spectral propagation network (SPN) that applies the spectral guided filter and channel attention mechanism to restore rich, and high-fidelity hyperspectral measurement of the dynamic physical world from limited spatiospectral snapshot. The proposed SPN is evaluated extensively using non-blind, blind and semi-blind experiments. For the proposed CHILD and other abundant datasets, our SPN outperforms the state-of-the-art methods greatly.

I. Introduction

Hyperspectral videography captures scenes containing abundant wavelength information for delineating a real-world scene representation, which is a persistent target in high dimensional sensing and computational photography. In various fields, such as agriculture, medicine, mineralogy, surveillance and environment [1], the hyperspectral images (HSIs) are beneficial to perceive the nature of objects for containing abundant information of the spatial-spectral distribution [2].