Video Super-Resolution Based on 3D-CNNS with Consideration of Scene Change

In video super-resolution, the spatio-temporal coherence between, and among the frames must be exploited appropriately for the accurate prediction of the high resolution frames. Although 2D-CNNs are powerful in modelling images, 3D-CNNs are more suitable for spatio-temporal feature extraction as they can preserve the temporal information. To this end, we propose an effective 3D-CNN for video super-resolution that does not require motion alignment as preprocessing. The proposed 3DSRnet maintains the temporal depth of spatio-temporal feature maps to maximally capture the temporally nonlinear characteristics between low and high resolution frames, and adopts residual learning in conjunction with the sub-pixel outputs. It outperforms the state-of-the-art method with average 0.45 dB and 0.36 dB higher in PSNR, for scale 3 and 4, in the Vidset4 benchmark. Our 3DSRnet first deals with the performance drop due to scene change, which is important in practice but has not been previously considered.

1. INTRODUCTION

Super-resolution (SR) methods upscale low resolution (LR) images or video frames to high resolution (HR) ones. One application of SR is in solving the lack of Ultra High Definition (UHD) visual content. Likewise, many SR applications are in videos, where the reconstruction of HR frames may benefit from additional information contained in the previous and future LR frames. This can be used to constrain the solution space of SR, which is an ill-posed problem. While video frames exhibit high temporal coherence, the camera or object motion can also provide a different angle or scale of the parts in the current frame in the consecutive surrounding frames, which can be effectively utilized as crucial clues in constructing high quality HR frames. Consequently, a video SR algorithm should fully exploit the temporal relations between the frames to aggregate them with the spatial information.