Scale Invariant Feature Transform Flow trajectory approach with applications to human action recognition

In this paper, we apply Scale Invariant Feature Transform (SIFT) Flow, a recently developed method of video representation to human action recognition. SIFT Flow provides a convenient way to express the displacement between keypoints, points which are invariant to scale changes spatially, in two adjacent frames of a video, and it furnishes a compact way to describe the behaviour at keypoints and their neighborhoods as they move in time. A dense trajectory approach using keypoints is developed, and its shape descriptor can be obtained. Local appearance descriptor like histogram of oriented gradients (HOG) evaluated at keypoints, local motion descriptors, like histogram of oriented flows (HOF) and motion boundary histogram (MBH) can be evaluated using SIFT flows. The HOG, HOF MBH, evaluated using SIFT flow, and the keypoint trajectory shape descriptor, together can be used as a feature vector to represent the video. We compare the performance of a number of classifiers to classify the feature vectors, including a bag-of-words approach, support vector machines, linear and nonlinear. It is shown that the proposed novel approach based on keypoints, and SIFT flows produces competitive results when compared with other state-of-the-art results.