Combining Pose and Trajectory for Skeleton Based Action Recognition using Two-Stream RNN

The changes of skeleton joint positions in skeleton sequences are caused by pose evolution and human body movements in the 3D space over time, which are both essential for action recognition. Most existing methods use relative coordinate system dependent on specific skeleton joint to track the action. However, because of the relative coordinate system,so the movements of action are eliminated. In this paper, the original coordinate system for raw skeleton sequences is transformed into the pose coordinate system and the trajectory coordinate system. Then the salient pose descriptor and trajectory descriptor of action sequence are extracted in the pose coordinate system and the trajectory system, respectively. The salient pose descriptor models the pose evolution over time while the salient trajectory descriptor describes the human body movements in the 3D space. Our method use a two-stream LSTM network to describe the changes of the temporal sequences. The salient pose and trajectory descriptors are fed into pose stream and trajectory stream separately. In order to merge the features from the pose stream and trajectory stream,we concatenate those features and use a fully connected network to de the next step. We evaluated our experiment on two benchmark datasets. Our experiment shows that our method gets good performance on two datasets.