An eXplainable Self-Attention-Based Spatial–Temporal Analysis for Human Activity Recognition

Human activity recognition (HAR) from inertial sensors has been a hot topic of research in recent years. Recognition of human activities from smartphone sensors poses a significant challenge because of the complexity of sensor data. In this work, a novel self-attention convolutional neural network-long short-term memory (CNN-LSTM) module is proposed by using the sensor data from inbuilt inertial sensors in smartphones. The experiment includes the results on the well-known HAR public datasets, namely, University of California HAR Dataset (UCI-HAR), University of California Human Activities and Postural Transitions (UCI-HAPT), and Sanitation datasets. The performance evaluation of the proposed model confirmed its effectiveness, as it achieved high accuracy rates across all tested datasets. Furthermore, ablation studies performed on the depth and number of hyperparameters used in the model demonstrate its effectiveness in achieving higher classification accuracy. The model was also evaluated using other metrics and was found to be cost effective in terms of the number of floating-point operations per second (FLOPs). In addition, a 1-D gradient-weighted class activation mapping (GradCAM) was implemented on the self-attention layer of the proposed model to determine and visualize the features responsible for the predicted activities. Our proposed model demonstrated superior performance on each individual dataset and achieved the best results on the UCI-HAR dataset. Specifically, for the UCI-HAR dataset, our model achieved an accuracy score of 0.9829, precision of 0.9833, recall of 0.9840, and
${F}1$
score of 0.9836 on the test data. Furthermore, the Matthews correlation coefficient (MCC) and Kappa score were calculated as 0.9795 and 0.9794, respectively.