Online Consistent Video Depth with Gaussian Mixture Representation

We demonstrate how off-the-shelf single-image depth estimation methods can be augmented with guidance from optical flow to achieve consistent and accurate online depth estimation using video sequences of static scenes. While previous work has successfully leveraged the complementary nature of optical flow and depth estimation, these techniques use computationally expensive test time optimization strategies that do not generalize beyond a single video sequence and also require knowledge of the future. In contrast, we present a computationally efficient feed-forward design that runs in an online fashion by utilizing learned data priors from previously seen video sequences. To accomplish this, we propose a continuous geometric scene representation that parametrically and compositionally represents the scene as a Gaussian Mixture Model (GMM). Based on this representation, our pipeline learns to estimate consistent depths and associated camera poses from video sequences of static scenes without direct supervision. Our online method achieves state-of-the-art results compared against offline methods that require all sequence frames.

I. Introduction

Estimating dense depth from an online sequence of monocular images, such as those coming from a video stream, is a crucial component of many computer vision applications. For example, dense depth estimates are useful for enabling special 3D effects in augmented reality type applications [33], or for various computational photography applications like geometrically consistent scene editing [43] or relighting [8]. Online estimation of monocular depth from video also has important uses in real-time applications, such as with self-driving vehicles that rely on consumer camera systems instead of or in addition to LiDAR and RADAR [10].