Global Shapes and Salient Joints Features Learning for Skeleton-Based Action Recognition

Global shapes and local joints are significant cues to learn skeleton representations for human action recognition. However, most shapes-based methods ignore local joints information and thus perform less well when recognizing similar actions. In this letter, we propose a two-stream method of global shapes and salient joints features learning to address the above issues. Firstly, the global sparse shapes modeling stream (GSSM) explores the global skeletal shapes in Kendall space and codes the nonlinear shapes by a sparse coding and dictionary learning method. Then, the salient enhancing joints generation stream (SEJG) explores local joints information by extracting discriminative salient joints and generates frame-enhancing action sequences with invariant length by a bilinear frame interpolation module. To improve the temporal modeling ability of our method, we use similar multi-scale LSTM methods in both GSSM and SEJG, which explore global-local features in static, short-term, and long-term temporal scales. Extensive experiments on four challenging datasets verify the effectiveness of our proposed method, which achieves competitive results compared to state-of-the-art (SOTA) methods.

I. Introduction

Human action recognition (HAR) is a central task in video understanding [1], [2], [3]. Existing studies have explored various modalities for feature extraction, such as optical flows [4], [5], [6], RGB images [7], [8], [9], and human skeletons [10], [11], [12]. Optical flows extract the temporal features immune to redundant environmental influence but leading to a relatively high computational cost. RGB images contain rich information but are sensitive to environmental noises. Human skeletons are action-focusing and compact but contain less information than optical flows and RGB images. Despite their sparsity, human skeletons have received increasing attention in recent research on action recognition due to their light weights and robustness to environmental variations.