MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

Motor Imagery-based Brain Computer Interface (MI-BCI) is a typical active BCI with a main focus on motor intention identification. Hybrid motor imagery (MI) decoding methods that based on multi-modal fusion of Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), especially deep learning-based methods, become popular in recent MI-BCI studies. However, the fusion strategy and network design in deep learning-based methods are complex. To solve this problem, we proposed the multi-channel fusion method (MCF) to simplify current fusion methods, and we designed a multi-channel fusion hybrid network (MCFHNet) based on MCF. MCFHNet combines depthwise convolutional layers, channel attention mechanism, and Bidirectional Long Short Term Memory (Bi-LSTM) layers, which enables strong capability of feature extraction in spatial and temporal domain. The comparison between MCFHNet and representative deep learning-based methods was performed on an open EEG-fNIRS dataset. We found the proposed method can yield superior performance (mean accuracy of 99.641 % in 5-fold cross validation of an intra-subject experiment). This work provides a new option for multi-modal MI decoding, which can be applied in the rehabilitation field based on hybrid BCI systems.

I. Introduction

Brain computer interface (BCI) is a system that enables communication between the brain and machines without the assistant of peripheral nerves and muscles [1]. MI - BCI is one of the most important and popular BCI paradigms. MI-BCI studies mainly focus on the MI decoding problem based on biological signals like EEG. Target limbs related to motor intention are identified by feature extraction and classification from biological signals. Currently, high-performance MI decoding is still the challenge in the MI-BCI field.