Unified Video Reconstruction for Rolling Shutter and Global Shutter Cameras

Currently, the general domain of video reconstruction (VR) is fragmented into different shutters spanning global shutter and rolling shutter cameras. Despite rapid progress in the state-of-the-art, existing methods overwhelmingly follow shutter-specific paradigms and cannot conceptually generalize to other shutter types, hindering the uniformity of VR models. In this paper, we propose UniVR, a versatile framework to handle various shutters through unified modeling and shared parameters. Specifically, UniVR encodes diverse shutter types into a unified space via a tractable shutter adapter, which is parameter-free and thus can be seamlessly delivered to current well-established VR architectures for cross-shutter transfer. To demonstrate its effectiveness, we conceptualize UniVR as three shutter-generic VR methods, namely Uni-SoftSplat, Uni-SuperSloMo, and Uni-RIFE. Extensive experimental results demonstrate that the pre-trained model without any fine-tuning can achieve reasonable performance even on novel shutters. After fine-tuning, new state-of-the-art performances are established that go beyond shutter-specific methods and enjoy strong generalization. The code is available at https://github.com/GitCVfb/UniVR.

I. Introduction

As a fundamental video processing task, the goal of video reconstruction (VR), is to generate the desired in-between frames given a pair of consecutive image frames [1], [2]. VR involves the understanding of pixel motion, image appearance, and even 3D structure, which contributes to many practical applications, such as slow-motion animation [3], [4], video compression [5], [6], novel view synthesis [7], [8], and other real-world systems [9], [10], [11], [12]. In recent years, a plethora of VR techniques has been actively studied around the common global shutter (GS) and rolling shutter (RS) cameras, e.g., GS video frame interpolation [3] and RS temporal super-resolution [13], with increasingly impressive results powered by the rapid progress of deep neural networks.