Controllable Space-Time Video Super-Resolution via Enhanced Bidirectional Flow Warping

Space-time video super-resolution targets to increase a given video's frame rate and resolution simultaneously. Al-though existing approaches have made great progress, most of them still suffer from the inaccurate approximation of large motions or fail to generate temporal consistent motion trajectory. To alleviate these problems, we carefully review the characteris-tics of different optical flow warping strategies, integrating and enhancing them to achieve more robust capabilities for handling extreme motions and time-modulated interpolation. Specifically, we utilize enhanced backward warping to perform alignment, mine space-time information across low resolution input frames, and propose an enhanced forward warping strategy to interpolate arbitrary intermediate frames. Furthermore, the proposed model can be trained end-to-end and produce intermediate results at any time by merely supervising the center moment. Experimental results show that the proposed algorithm performs favorably against the state-of-the-art methods in objective metrics and subjective visual effects.

I. Introduction

The purpose of space-time video super-resolution (ST-VSR) is to convert video with low-frame-rate (LFR) and low resolution (LR) to higher temporal and spatial resolutions. With the continuous advancement of high definition playback equipment, high-resolution slow-motion video sequences are becoming more and more popular among the public. To increase the spatio-temporal resolution of a given video, earlier traditional method [1] consider this task as an optimization problem and usually relies on strong assumptions or prior knowledge consequently hard to apply to complex and diverse scenarios. In reality, space-time video super-resolution can be divided into two sub-tasks: video frame interpolation (VFI) and video super resolution (VSR). For VFI, Some flow-based methods [2]–[4] employ optical flow as motion guidance information to approximate the intermediate frame. Other kernel-based methods[5], [6] leverage adaptive convolution for interpolation. For VSR, most restoration algorism can be divided into two categories: temporal sliding window-based methods [7]–[9] and recurrent methods [10], [11]. The research development of VFI and VSR also promotes the progress of ST-VSR, which is more efficient than the two-stage approaches. Specifically, STARnet[12] first computes the optical flow of two adjacent frames and then applies feature warping to synthesize the intermediate frame. Xiang et al. [13] propose a deformable alignment structure and adopt a bidirectional convLSTM network to leverage preceding and succeeding information from the whole input sequence. Based on [13], Xu et al. further propose TMNet [14] which can perform controllable frame interpolation at any intermediate moment. Despite the remarkable progress of the aforementioned methods, they still suffer from pixel misplacement when handling extreme motions.