Multi-view scene capture by surfel sampling: from video streams to non-rigid 3D motion, shape and reflectance

In this paper we study the problem of recovering the 3D shape, reflectance, and non-rigid motion of a dynamic 3D scene. Because these properties are completely unknown, our approach uses multiple views to build a piecewise continuous geometric and radiometric representation of the scene's trace in space-time. Basic primitive of this representation is the dynamic surfel, which (1) encodes the instantaneous local shape, reflectance, and motion of a small region in the scene, and (2) enables accurate prediction of the region's dynamic appearance under known illumination conditions. We show that complete surfel-based reconstructions can be created by repeatedly applying an algorithm called surfel sampling that combines sampling and parameter estimation to fit a single surfel to a small, bounded region of space-time. Experimental results with the Phong reflectance model and complex real scenes (clothing, skin, shiny objects) illustrate our method's ability to explain pixels and pixel variations in terms of their physical causes-shape, reflectance, motion, illumination, and visibility.

1. Introduction

In this paper we consider the problem of Multi-View Scene Capture-using multiple cameras to simultaneously recover the shape, reflectance and non-rigid motion of an unknown scene that evolves through time in a completely unknown way. While many techniques exist for recovering one of these properties when the rest of them are known (e.g., capturing the 3D motion of articulated [1], or deformable scenes [2]; reconstructing static Lambertian scenes [3], [4]; and recovering the reflectance of static scenes with known shape [5], [6]), our focus here is on the general case. In particular, how can we capture 3D scenes whose appearance depends on time-varying interactions between shape, reflectance, illumination, and motion? Answering this question would go a long way toward reconstructing many common real-world scenes that are beyond the current state of the art, including (1) highly-deformable and geometrically-complex surfaces whose shape, motion, self-occlusions and self-shadows change through time (e.g., clothing [7]), (2) non-Lambertian surfaces with complex shape and deformation properties (e.g., mm-scale dynamic representations of the human body), and (3) static or moving 3D objects with specular surfaces [8].