Scene flow by tracking in intensity and depth data

The scene flow describes the motion of each 3D point between two times steps. With the arrival of new depth sensors, as the Microsfot Kinect, it is now possible to compute scene flow with a single camera, with promising repercussion in a wide range of computer vision scenarios. We propose a novel method to compute scene flow by tracking in a Lucas-Kanade framework. Scene flow is estimated using a pair of aligned intensity and depth images, but rather than computing a dense scene flow as in most previous methods, we get a set of 3D motion fields by tracking surface patches. Assuming a 3D local rigidity of the scene, we propose a rigid translation flow model that allows to solve directly for the scene flow by constraining the 3D motion field both in intensity and depth data. In our experimentation we achieve very encouraging results. Since this approach solves simultaneously for the 2D tracking and for the scene flow, it can be used for action recognition in existing 2D tracking based methods or to define scene flow descriptors.

1. Introduction

The scene flow corresponds to the 3D motion field of the scene [31] and since it provides the motion of 3D points it can be used in several tasks, such as object segmentation, motion analysis and tracking. In particular, in human activity understanding the scene flow can be useful to pro-vide features for classification and recognition algorithms. However, there is not work that directly compute scene flow and perform tasks like human action recognition. Probably, this is due to the fact that most existing methods compute a dense scene flow, spending a lot of processing time, and they require a stereo or multi-view camera systems which are not always available. On the other hand, the optical flow, that corresponds to the scene flow projection onto the image plane, has been successfully used in human action recognition. Optical flow is commonly used in state of the art techniques as part of descriptors formed by histograms [17], [26] or computed to extract 2D trajectories [23], [22], [27].