Multimodal Fusion Convolutional Neural Network Based on sEMG and Accelerometer Signals for Intersubject Upper Limb Movement Classification

The variation in the distributions of recorded data between individuals leads to low classification accuracy. To address this issue, we introduce a multimodal fusion convolutional neural network (MFCNN). This network extracts common information from surface electromyography (sEMG) and accelerometer signals of different subjects using a two-stream convolutional neural network (CNN). To enhance the classification accuracy of a particular subject, a fine-tuning approach was implemented. The performance of the proposed method was assessed in four different scenarios, which include intersubject classification, intersubject classification when training data from multiple subjects, fine-tuned intersubject classification, and fine-tuned intersubject classification when training data from multiple subjects. The results demonstrate that in the intersubject scenario, when multiple subjects are available for training, the MFCNN achieves higher classification accuracy (
${p} < 0.05$
) than other neural networks and support vector machines (SVMs) that use sEMG signals [neural network (NN) and SVM], accelerometer signals (accNN and accSVM), sEMG and accelerometer signals [multimodal fusion nerual network (MFNN) and multimodal fusion support vector machine (MFSVM)] as inputs, as well as a CNN that uses sEMG signals as input after fine-tuning. Furthermore, compared with an MFCNN model trained with data from a single subject and an accCNN model trained with data from a single subject or multiple subjects, an MFCNN trained with multiple subjects demonstrated better performance on new subjects after fine-tuning (
${p} < 0.05$
). This method can learn common features among different subjects and improve the performance of classification among subjects. Our proposed method demonstrates the innovation of using a multimodal fusion approach and two-stream CNN to improve intersubject classification accuracy in upper limb movements.