MMHAR-EnsemNet: A Multi-Modal Human Activity Recognition Model

In this article, we propose a new deep learning model named as MMHAR-EnsemNet (Multi-Modal Human Activity Recognition Ensemble Network) which makes use of four different modalities to perform sensor-based Human Activity Recognition (HAR).Two separate Convolutional Neural Networks (CNNs) are made for skeleton data. While one CNN and one LSTM is trained for RGB images. For Accelerometer and Gyroscope data first it is converted to signal diagram then another CNN model is trained. Finally, all the outputs of the said models have been used to form an ensemble so that performance of the HAR model gets improved. The proposed model has been evaluated on two standard benchmark datasets namely UTD-MHAD and Berkeley-MHAD which contain four different modalities of input information. Experimental results confirm that the MMHAR-EnsemNet model has outperformed some recently proposed models considered here for comparison. Source code of this work can be found at: https://github.com/abhi1998das/MMHAREnsemNet.