Space-Time Behavior-Based Correlation-OR-How to Tell If Two Underlying Motion Fields Are Similar Without Computing Them?

We introduce a behavior-based similarity measure that tells us whether two different space-time intensity patterns of two different video segments could have resulted from a similar underlying motion field. This is done directly from the intensity information, without explicitly computing the underlying motions. Such a measure allows us to detect similarity between video segments of differently dressed people performing the same type of activity. It requires no foreground/background segmentation, no prior learning of activities, and no motion estimation or tracking. Using this behavior-based similarity measure, we extend the notion of two-dimensional image correlation into the three-dimensional space-time volume and thus allowing to correlate dynamic behaviors and actions. Small space-time video segments (small video clips) are "correlated" against the entire video sequences in all three dimensions (x, y, and t). Peak correlation values correspond to video locations with similar dynamic behaviors. Our approach can detect very complex behaviors in video sequences (for example, ballet movements, pool dives, and running water), even when multiple complex activities occur simultaneously within the field of view of the camera. We further show its robustness to small changes in scale and orientation of the correlated behavior.

1 Introduction

Different people with similar behaviors induce completely different space-time intensity patterns in a recorded video sequence. This is because they wear different clothes, and their surrounding backgrounds are different. What is common across such sequences of the same behaviors is the underlying induced motion fields. This observation was used in [9], where low-pass filtered optical-flow fields (between pairs of frames) were used for action recognition. However, dense, unconstrained, and nonrigid motion estimation is highly noisy and unreliable. Clothes worn by different people performing the same action often have very different spatial properties (different color, texture, and so forth). Uniform-colored clothes induce local aperture effects, especially when the observed acting person is large (which is why Efros et al. [9] analyze small people "at a glance"). Dense flow estimation is even more unreliable when the dynamic event contains unstructured objects like running water, flickering fire, and so forth.