Enhancing Video Super-Resolution via Implicit Resampling-based Alignment

In video super-resolution, it is common to use a frame-wise alignment to support the propagation of information over time. The role of alignment is well-studied for low-level enhancement in video, but existing works overlook a critical step - resampling. We show through extensive experiments that for alignment to be effective, the resam-pIing should preserve the reference frequency spectrum while minimizing spatial distortions. However, most ex-isting works simply use a default choice of bilinear inter-polation for resampling even though bilinear interpolation has a smoothing effect and hinders super-resolution. From these observations, we propose an implicit resampling-based alignment. The sampling positions are encoded by a sinusoidal positional encoding, while the value is es-timated with a coordinate network and a window-based cross-attention. We show that bilinear interpolation inher-ently attenuates high-frequency information while an MLP-based coordinate network can approximate more frequen-cies. Experiments on synthetic and real-world datasets show that alignment with our proposed implicit resampling enhances the performance of state-of-the-art frameworks with minimal impact on both compute and parameters.

1. Introduction

Video super-resolution (VSR) recovers a high spatial reso-lution sequence of frames from a low-resolution sequence. While image super-resolution can be applied naively to each frame individually, the temporal correlations across the frames give an extra source of information to im-prove the super-resolved output. As such, the main differ-ence in video versus image super-resolution architectures lies in the use of temporal dependencies. Previous works [2], [9], [2]6, [2]8 have shown that spatial alignment is an essen-tial pre-processing step for effective information exchange across the frames. Given the frame-to-frame camera and object motions, alignment provides indications of sub-pixel information which can benefit the super-resolution.