sEMG and IMU Data-Based Hand Gesture Recognition Method Using Multistream CNN With a Fine-Tuning Transfer Framework

Interfaces based on surface electromyography (sEMG) and inertial measurement units (IMUs) enable users to interact with computers in a natural and intuitive way through hand gestures. sEMG or IMU sensor-based approaches can provide muscle electrical activity or motion information to recognize gestures. However, the existing methods recognize static and dynamic gestures separately and hierarchically. In addition, the variability in sEMG data among different subjects limits the performance of sensor-based human–computer interfaces. To address these limitations, this article proposes a multistream convolutional neural network (CNN) architecture with a fine-tuning transfer strategy. The multistream architecture explores the complementary nature of sEMG and IMU signals and achieves real-time recognition of dynamic and static gestures in a nonhierarchical way. In addition, a new dataset including seven static gestures and six dynamic gestures is designed for evaluation. The proposed method is verified by experiments on six subjects and is compared with previous methods on the same dataset. Experimental results reveal that the transfer strategy significantly improves recognition accuracy from 89.99% to 98.49%. Meanwhile, by learning the fusion features of sEMG and IMU signals, the recognition accuracy is enhanced from 49.54% (sEMG) and 59.91% (IMU) to 89.99%. In addition, the accuracy and latency of this method for real-time recognition are 98.12% and 103 ms, respectively. These results demonstrate that the proposed multistream CNN model accurately recognizes these static and dynamic gestures online in a nonhierarchical way and can effectively utilize the complementarity between sEMG and IMU signals.