Lightweight graph convolutional networks combining multimodal fusion and spatio-temporal attention mechanisms

Since most mainstream human action recognition methods suffer from high operational complexity and large network parameters, and skeleton behavior recognition is inaccurate for spatio-temporal feature extraction and difficult to capture global contextual information. Firstly, in the data pre-processing stage, three kinds of information data streams related to nodes and skeleton are fused so as to reduce the number of network parameters and improve the operation speed; secondly, the recognition accuracy is improved by constructing a spatio-temporal attention module based on non-local operations to focus on nodes with action discriminative; then a spatio-temporal feature extraction module is constructed to obtain spatio-temporal association information; finally, the Softmax layer is used for action The final action recognition is performed by the Softmax layer. The algorithm achieves accuracies of 89.4% and 95.5% for the two evaluation metrics on the NTU60 RGB+D dataset, and 82.4% and 82.8% on the NTU120 RGB+D dataset. Finally, we did a comparison test with other methods on the dataset UCF101, and the results showed that the recognition speed and accuracy were improved, reflecting the superiority and reliability of the method.