Human Activity Recognition Based on Radar and Video Surveillance Sensor Fusion

Activity monitoring is an actual problem in the prevention of life-threatening conditions. This question is especially important for people over 65 who fall at least once a year. Unstable gait is one of the predictors of falls and the future progression of cognitive diseases. Hence, abnormal gait detection can potentially be a useful tool for the prevention of falls and the diagnosis of cognitive impairment. In this paper, a sensor fusion approach for human activity recognition is presented. Radar and video surveillance data of eight volunteers (19–37 years old) were used for training the deep learning model for the binary classification problem (normal/unstable gait). Radar data were processed using a wavelet transform, and a pre-trained AlexNet neural network was fitted using a training dataset. Information about skeletal key points was extracted from video frames by means of a BlasePose neural network and used for training the recurrent neural network with long-short-term memory units. Then, a sensor fusion approach at the classification level was applied. The accuracy and Cohen's kappa of the radar-only model were 0.931 and 0.859, respectively. The accuracy and Cohen's kappa of the video-only model were 0.966 and 0.930, respectively. Finally, the accuracy and Cohen's kappa for the sensor fusion-based model were 0.931 and 0.859, respectively. The results of our work can be potentially used for the design of the system for human activity recognition in the smart home technology paradigm.