Improving Video Super-Resolution with Long-Term Self-Exemplars

Existing video super-resolution methods often utilize a few neighboring frames to generate a higher-resolution image for each frame. However, the abundant information in distant frames has not been fully exploited in these methods: corresponding patches of the same instance appear across distant frames at different scales. Based on this observation, we propose to improve the video super-resolution quality with long-term cross-scale aggregation that leverages similar patches (self-exemplars) across distant frames. Our method can be implemented as post-processing for any super-resolution methods to improve performance. Our model consists of a multi-reference alignment module to fuse the features derived from similar patches: we fuse the features of distant references to perform high-quality super-resolution. We also propose a novel and practical training strategy for reference-based super-resolution. To evaluate the performance of our proposed method, we conduct extensive experiments on our collected CarCam dataset, the Waymo Open dataset, and the REDS dataset, and the results demonstrate our method outperforms state-of-the-art methods.