Multimodal recognition of speech and electrocorticogram

Brain-Computer Interface (BCI) provides a novel way of communicating with computers but their poor performance in terms of speed and accuracy in comparison to the other modes of communication has been the biggest obstacle in their usage for practical applications. In this work, we aim to enhance the performance of BCI by utilizing speech data collected along with the electrocorticogram (ECoG) recordings when the person is speaking. While some BCI users may have difficulty in speaking at all, many of them can speak, even though their speech may be unclear. We propose that information from this speech data can help in improving accuracy and to employ such speech context to assist the decoding process, we apply a multimodal recognition method. We use speech data contaminated by noise in our evaluation to simulate the cases where the available speech quality is low. Our experiments using data from five subjects suffering from Epilepsy show that our method of using multimodal input has a significant improvement, an absolute reduction in phrase error rate by 1.1 points from recognition using speech alone and by 51.3 points from recognition using ECoG alone.