A 3D-CLDNN Based Multiple Data Fusion Framework for Finger Gesture Recognition in Human-Robot Interaction

Finger gesture recognition using surface electromyography (sEMG) became an efficient Human-Robot Interaction (HRI) solution. Although Machine Learning (ML) techniques are widely applied in this field, the general solutions for labeling and collecting big datasets impose time-consuming implementation and heavy workloads. In this paper, a new deep learning structure, namely three-dimensional convolutional long short-term memory neural networks (3D-CLDNN) for finger gesture identification based on depth vision and sEMG signals, was proposed for human-machine interaction. It automatically labels the depth data by the self-organizing map (SOM) and predicts the hand gesture only adopting sEMG signals. The 3D-CLDNN method is integrated to improve the recognition rate and computational speed. The results showed the highest clustering accuracy (98.60%) and highest accuracy (84.40%) with the lowest computational time compared with different approaches. Finally, real-time human-machine interaction experiments are performed to demonstrate its efficiency.