Super-resolved Chromatic Mapping of Snapshot Mosaic Image Sensors via a Texture Sensitive Residual Network

This paper introduces a novel method to simultaneously super-resolve and colour-predict images acquired by snapshot mosaic sensors. These sensors allow for spectral images to be acquired using low-power, small form factor, solid-state CMOS sensors that can operate at video frame rates without the need for complex optical setups. Despite their desirable traits, their main drawback stems from the fact that the spatial resolution of the imagery acquired by these sensors is low. Moreover, chromatic mapping in snapshot mosaic sensors is not straightforward since the bands delivered by the sensor tend to be narrow and unevenly distributed across the range in which they operate. We tackle this drawback as applied to chromatic mapping by using a residual channel attention network equipped with a texture sensitive block. Our method significantly outperforms the traditional approach of interpolating the image and, afterwards, applying a colour matching function. This work establishes state-of-the-art in this domain while also making available to the research community a dataset containing 296 registered stereo multi-spectral/RGB images pairs.

1. Introduction

Imaging spectroscopy devices can capture an information-rich representation of a scene, often in terms of tens or hundreds of wavelength-indexed bands. Recent advances in imaging spectroscopy have seen the development of real-time snapshot mosaic image sensors, which are compact in size and exhibit comparable frame rates to current trichromatic cameras [44], [6]. Despite of the extensive interest in snapshot mosaic sensors and their potential for multi-spectral imaging, they suffer from an inherent trade-off between the spatial and spectral resolution. This is as a result of their architecture, where the raw resolution of the detector is distributed across the number of wavelength-indexed bands in the spectral image produced at output.