3-D Human Behavior Understanding Using Generalized TS-LSTM Networks

This paper addresses the problems of skeleton feature representation and the modeling of temporal dynamics to recognize human actions consisting of poses. In contrast to traditional methods which generally used relative coordinate systems dependent on some joints, or modeled only the long-term dependency, we attempt to understand 3D human behavior with observation by taking temporally different windows. Instead of taking raw skeletons as the input, we transform the skeletons into another coordinate system to obtain the robustness to scale, rotation and translation, and extract motion features between adjacent skeletons, which finally constructs an efficient hybrid-stream combining both pose and motion streams. We propose novel generalized Temporal Sliding Long Short-term Memory (TS-LSTM) networks. The proposed networks are composed of multiple TS-LSTM networks with various hyper-parameters, which can capture various temporal dynamics of actions. We also propose a novel hyper-parameter searching method, which finds decent hyper-parameters of generalized TS-LSTM to handle temporal dynamics of actions. In the experiment, we evaluate the proposed networks to verify the effectiveness of the proposed methods, and compare them with the other methods on three challenging datasets. Additionally, we analyze a relation between the recognized actions and the hyper-parameters, and visualize the layers of the proposed models.