3D Deformable Kernels for Video super-resolution

Video super-resolution are drawing increasing attention in the computer vision community. Temporal modeling is crucial for video super-resolution. A challenge for video super-resolution to fully mining temporal-spatial information in video sequence. In this work, we propose 3D deformable kernels for video super-resolution (DK3Dnet). Specifically, we introduce 3D deformable kernels (DK3D) to integrate deformable convolution with 3D convolution to enhance spatio-temporal modeling capability. To enhance the quality of subsequent restoration. we use a Temporal and Spatial Attention fusion module (TSA fusion), in which attention is applied both temporally and spatially. Finally, we use channel-wise attention residual block (CARB) to enhance the quality of video frame in DK3Dnet reconstruction module. Experimental results show that DK3Dnet can exploiting spatio-temporal information to improve the performance of video super-resolution.

1. Introduction

Super-resolution (SR) is an essential visual task with the aim of creating a high-resolution (HR) image based on a low-resolution image (LR) by compensating for the missing details in the low-resolution image. Video super-resolution (VSR) is an important embranchment of research in super-resolution methods. Since the video input is composed of consecutive frames, the correlation between video frames is particularly important for the performance of video super-resolution. In recent years, due to the breakthroughs in convolutional neural networks (CNN), CNN-based VSR methods have performed better than traditional VSR methods in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity index (SSIM) and other evaluation metrics.