3D Skeleton-Based Human Motion Prediction Using Dynamic Multi-Scale Spatiotemporal Graph Recurrent Neural Networks

A dynamic multi-scale spatiotemporal graph recurrent neural network (DMST-GRNN) model has been introduced, which is leveraged to use human motion prediction on a 3D skeleton-based human activity dataset. It offers a multi-scale approach to spatial & temporal graphs using multi-scale graph convolution units (MGCUs) to describe the human body's semantic interconnection. The proposed DMST-GRNN is an encoder-decoder framework where a series of MGCUs are used as encoders to learn spatiotemporal features and a novel graph-gated recurrent unit lite (GGRU-L) for the decoder to predict human pose. Extensive experiments have been carried out with two datasets, Human3.6M and CMU Mocap, where both short and long videos were considered to validate the performance of the proposed model. The DMST-GRNN model outperforms the existing baseline on the Human3.6M datasets by 11.95% and 7.74% of average mean angle errors (avg MAE) for short and long-term motion prediction, respectively. Similarly, CMU Mocap datasets, the DMST-GRNN model predicts future posture more accurately than the previous best approaches by 2.77% and 5.51% of average mean angle errors (avg MAE) for short and long-term motion prediction, respectively. A comparison analysis was also presented with other measures like mean angle error, prediction loss and standard deviation. A separate discussion has been included to analyze the effect of different multiscale on spatial and temporal graphs, along with the impact of MGCU unit counts.

I. Introduction

Nowadays human body 3D skeleton is used to estimate the pose of the human body based on the previous activity. Human motion prediction is becoming increasingly popular since it enables computers to comprehend human activity. Such a strategy (skeleton-based human motion prediction) might be used in various computer vision and robotics applications, including human-computer interaction, autonomous driving, and pedestrian tracking [1], [2]. Real-world human motion prediction is challenging since the movement is unpredictable, flexible, and non-linear in nature. Many approaches have been put forward to address this problem, such as the traditional state-based and deep-network approaches, which provide group posture features to learn the motion sequence [3]. Although these approaches came up with satisfactory results and tried to handle inherent challenges, studying spatial or temporal relationships among the human body joints in this domain is still an opportunity. The temporal relationships represent inter-frame interaction between displaying the continuous movements, while the spatial relations record the underlying pose. Spatial features are to be calculated by determining the spatial deviation of each frame of the activity sequence. The temporal feature is associated with or changes over time. So, it seems promising to estimate the temporal change of each frame of the activity sequence.