Multi-Branch Networks for Video Super-Resolution With Dynamic Reconstruction Strategy

Recently, the rapid development of 2-dimensional (2D) convolutional neural networks (CNNs) has driven single image super-resolution (SISR) into a new era, owing to their powerful ability in modeling spatial relation within one single image. However, few studies focus on video super-resolution (VSR) due to the key challenge that apart from the spatial relation, the temporal dependence among consecutive low-resolution (LR) frames must be taken into consideration for better reconstruction. In this article, unlike most previous methods based on optical flow for motion compensation, 3-dimensional (3D) convolution is utilized to capture the temporal relation. Firstly, in contrast to the conventional 3D convolution which is notorious for the excessively high computational burden, we propose an efficient 3D convolutional block (E3DB) through convolution factorization principle (CFP), which significantly reduces the computing load while maximally maintaining the temporal information. Then, by taking advantage of E3DB, we propose a novel multi-resolution extraction block (MREB) which aggregates the information from multiple resolutions, leading to the stronger high-resolution representation learning and better feature extraction. Besides, based on our 3-branch architecture, instead of the simple addition or concatenation, a dynamic reconstruction strategy (DRS) is proposed to adaptively fuse the optimal information of temporal dependence from each branch. It is therefore termed as dynamic multiple branch network (DMBN). Comprehensive experiments on public benchmark datasets demonstrate the superiority of our DMBN over the current state-of-the-art methods in terms of accuracy and efficiency.

I. Introduction

Super-resolution (SR) is a classical problem that transforms low-resolution (LR) images to high-resolution (HR) counterparts, which has attracted increasing attention among computer vision community over past years. Despite the fact that SR is a hot issue in image processing, compared with single image super-resolution (SISR), video super-resolution (VSR) obtains less attention and it is more complicated. The goal of VSR is to reconstruct HR video frames from consecutive LR frames. Thus, besides the spatial relation, the additional temporal correlations among multiple LR inputs require to be exploited in order to obtain high-quality reconstruction results. Along with flourishing of 2D CNNs, which shows the remarkable ability in modeling spatial relation within one single image, the methods based on CNNs [1]–[3] for SISR have achieved promising performances. However, a great quantity of works [4], [5] have proved that directly applying the SISR network to solve the VSR problem usually yields suboptimal results. Thus, a vital issue in VSR is to effectively exploit temporal redundancies.