Deep Motion Regularizer for Video Snapshot Compressive Imaging

Video snapshot compressive imaging (SCI) samples 3D high-speed video frames with temporally varying spatial modulation and compresses them into a single 2D measurement, and the SCI reconstruction algorithm aims to recover the original high-speed frames from the measurement. However, conventional video SCI systems encounter challenges when raising the frame rate of the reconstructed video. To achieve higher frame rates, the modulation mask's rate must be increased, which in turn leads to an increase in the associated hardware expenses. In this paper, we propose a deep unfolding-based reconstruction framework with optical flow for video SCI. The framework recovers both observed and unobserved frames from measurements, resulting in increased video frame rate. To estimate the optical flow, we transform the video features of the network into optical flow features during the iteration. Then, we design a deep denoiser and an optical flow-based motion regularizer combined with the voxels of coarse reconstructed frames to update the observed and unobserved frames. To improve the performance, we employ group convolution in the network and fuse the optical flow information from different phases to reduce the information loss. We further extend the proposed deep unfolding framework to the reconstruction of color SCI videos. Extensive experiments on benchmark data and real data prove that our proposed method has state-of-the-art reconstruction performance and visual effects.

I. Introduction

Video snapshot compressive imaging (SCI) is a promising computational imaging technique that has gained much attention in recent years [1], [2], [3], [4]. In the video SCI system, temporally varying spatial modulation is used to sample the continuous high-speed frames and compress them into a single measurement [5], [6]. There are several variants of modulation implementations, such as coded aperture compressive temporal imaging (CACTI) [5], digital micromirror device (DMD) [7], liquid crystal on silicon (LCOS) and hybrid modulation [8]. Using a suitable reconstruction algorithm, the original frames can be recovered from the compressed measurement. SCI thus has three major advantages: first, it boasts fast collection speeds without the need for expensive high-speed video capturing hardware. Second, SCI has low complexity due to its simple modulating mask design, which reduces the likelihood of mechanical failures and costly repairs. Third, SCI has low storage requirements since the compressed measurement obtained by SCI requires less storage space than traditional video data. This feature is especially valuable for applications such as unmanned aerial vehicles and space exploration, where storage and transmission of large amounts of data can be challenging [9], [10], [11], [12]. Overall, video SCI is a prospective imaging system that offers fast collection speed, low complexity, and low storage over traditional video acquisition syste ms requirements, making it a viable option for various applications.