Jointly Learning Band Selection and Filter Array Design for Hyperspectral Imaging

A single-shot multispectral camera equipped with an optimized color filter array (CFA) has the potential to deliver a fast and low-cost hyperspectral (HS) imaging system. Previous solutions are largely restricted to designing demosaicing algorithms for fixed CFAs – be it the Bayer color pattern or evenly-spaced spectral multiplexing patterns. Since sampling and reconstruction are tightly-coupled, the ability to search for an optimal solution is severely constrained by using predefined CFAs. In this work, we simultaneously address the problem of spectral band selection, CFA design, image demosaicing, and spectral image recovery in a joint learning framework for single-shot HS imaging. We propose a reinforcement learning (RL) based method for spectral band selection and a novel neural network for CFA generation, image demosaicing, and HS image recovery. The final spectral reconstruction accuracy is used to supervise the training of the main network to maximize the synergies between those tightly-related tasks. The RL method regards the main network as an agent to collect reward. Our final method delivers a simple setup – as simple as an RGB camera – for HS imaging. Experimental results show that our method outperforms competing methods by a large margin.