Hyperspectral Image Super-Resolution Based on Multi-Scale Wavelet 3D Convolutional Neural Network

Super-resolution (SR) of hyperspectral image (HSI) is of significance for its applications. Wavelet decomposition can be used to capture textures and structures in the HSI. In this study, we propose a multi-scale wavelet 3D convolutional neural network (MW-3D-CNN) for HSI SR. Instead of reconstructing the high resolution (HR) HSI directly, we predict the wavelet coefficients of HR HSI with the proposed network, which is composed of an embedding subnet and a predicting subnet. Both of them are built with 3D convolutional layers. The embedding subnet extracts deep spatial-spectral features from the low resolution (LR) HSI and represents the LR HSI as a set of feature cubes. The feature cubes are then fed to the predicting subnet. There are multiple output branches in the predicting subnet, each of which corresponds to a wavelet sub-band and predicts the wavelet coefficients of HR HSI. By applying inverse wavelet transform to the predicted wavelet coefficients, the HR HSI can be obtained. In the training stage, we propose to train MW-3D-CNN with L1 norm loss, which is more suitable than the conventional L2 norm loss for penalizing the errors in different wavelet sub-bands. In the experiment, the performance is tested on several HSI datasets.