Video Super Resolution Based on Deep Convolution Neural Network With Two-Stage Motion Compensation

In this paper, we propose methodologies to train highly accurate deep convolutional neural networks (CNNs) for video super resolution (SR). To use the inter-frame characteristic, we introduce a video SR network based on two-stage motion compensation (VSR-TMC). Firstly, the low resolution (LR) frames are aligned by the LR optical flow, and fed into a 3D-convolution network for spatial super resolution. This 3D-conv network generates the intermediate high resolution (HR) frames based on the aligned LR frames. The HR optical flow between the intermediate HR frames is further utilized to refine these HR frames as the final output. This HR optical flow could be estimated either from intermediate HR frames, or by a proposed super resolution network specifically working in the optical flow domain. Such optical flow SR network allows us to get the HR optical flow directly from the LR optical flow, which is more efficient compared to calculating HR optical flow from HR frames. Experimental results on publicly available dataset demonstrate that the VSR-TMC is significantly better compared to single image SR networks and video SR networks with LR motion compensation only. It achieves the state-of-the-art performance.

1. Introduction

Video super resolution (VSR) aims to retrieve a high resolution (HR) video based on the inputs from a low resolution (LR) video. It is widely used in many computer vision applications such as surveillance, satellite imaging, and video zooming. This problem is challenging because video sequences vary in both temporal and spatial dimensions. In contrast to single image super resolution (SISR) that the details can rely on external examples, a good video SR system needs considering the temporal consistency as well. In general, a video SR system consists of two parts, the temporal alignment module which applies motion compensation to keep the temporal consistency of the output video, and the spatial super resolution module which aims to retrieve the HR texture from the LR input.