Three-dimensional scene flow

Just as optical flow is the two-dimensional motion of points in an image, scene flow is the three-dimensional motion of points in the world. The fundamental difficulty with optical flow is that only the normal flow can be computed directly from the image measurements, without some form of smoothing or regularization. In this paper, we begin by showing that the same fundamental limitation applies to scene flow; however, many cameras are used to image the scene. There are then two choices when computing scene flow: 1) perform the regularization in the images or 2) perform the regularization on the surface of the object in the scene. In this paper, we choose to compute scene flow using regularization in the images. We describe three algorithms, the first two for computing scene flow from optical flows and the third for constraining scene structure from the inconsistencies in multiple optical flows.

1 Introduction

One of the fundamental properties of any scene is its motion. When a dynamically changing scene is imaged by a video camera, the optical flow between two neighboring frames tells us something about the motion of the scene. However, optical flow is just a two-dimensional motion field in the image plane. It is the projection of the three-dimensional motion of the world. If the world is completely nonrigid, the motions of the points in the scene may all be independent of each other. The scene motion is therefore a dense three-dimensional vector field defined for each point on every surface in the scene. By analogy with optical flow, we refer to this three-dimensional motion field as the scene flow [22].