ViTA: Video Transformer Adaptor for Robust Video Depth Estimation

Depth information plays a pivotal role in numerous computer vision applications, including autonomous driving, 3D reconstruction, and 3D content generation. When deploying depth estimation models in practical applications, it is essential to ensure that the models have strong generalization capabilities. However, existing depth estimation methods primarily concentrate on robust single-image depth estimation, leading to the occurrence of flickering artifacts when applied to video inputs. On the other hand, video depth estimation methods either consume excessive computational resources or lack robustness. To address the above issues, we propose ViTA, a video transformer adaptor, to estimate temporally consistent video depth in the wild. In particular, we leverage a pre-trained image transformer (i.e., DPT) and introduce additional temporal embeddings in the transformer blocks. Such designs enable our ViTA to output reliable results given an unconstrained video. Besides, we present a spatio-temporal consistency loss for supervision. The spatial loss computes the per-pixel discrepancy between the prediction and the ground truth in space, while the temporal loss regularizes the inconsistent outputs of the same point in consecutive frames. To find the correspondences between consecutive frames, we design a bi-directional warping strategy based on the forward and backward optical flow. During inference, our ViTA no longer requires optical flow estimation, which enables it to estimate spatially accurate and temporally consistent video depth maps with fine-grained details in real time. We conduct a detailed ablation study to verify the effectiveness of the proposed components. Extensive experiments on the zero-shot cross-dataset evaluation demonstrate that the proposed method is superior to previous methods.

I. Introduction

Temporally consistent video depth is essential in many vision-related tasks, such as video editing effects [1], [2], [3], bokeh rendering [4], [5], and 2D-to-3D conversion [6], [7]. Although recovering depth from a monocular video has been studied for decades, it is still challenging to estimate spatially accurate and temporally consistent depth for unconstrained videos. Such videos can be any monocular video, captured in indoor or outdoor scenes, with a stationary or moving camera, and may contain moving objects or not. Previous methods can work in specific conditions but show poor generalization to such casual videos.