Bimodal Fusion Network for Basic Taste Sensation Recognition from Electroencephalography and Electromyography

Taste sensation can be objectively measured using electroencephalography (EEG) or electromyography (EMG). How-ever, it is still challenging to effectively utilize the complementary information from EEG and EMG signals in taste sensation recognition. This paper proposes a bimodal fusion network (Bi-FusionNet) for recognizing basic taste sensations (sour, sweet, bitter, salty, umami, and blank). Two convolutional backbones with similar structures are designed to separately extract the single-modal features of EEG and EMG. Then, EEG and EMG features are concatenated for bimodal interaction and complementarity. Finally, three loss functions are adopted: a center loss for aggregating intra-class samples, a mean squared error loss for sequence positions for minimizing the difference between signals during the stimulation, and a softmax loss for minimizing the entropy of prediction and true labels. The results on the taste sensation dataset show that bimodal fusion improves recognition performance, and Bi-FusionNet outperforms single-modal methods and other fusion methods. Bi-FusionNet paves the way for the application of multimodal fusion in taste sensation recognition.

1. INTRODUCTION

Taste is an important sensory for humans to ensure safe ingestion and avoid harmful substances. However, taste diminishes or even disappears with age, and taste disorder is also one of the common concomitants of coronavirus dis-ease 2019 (COVID-19) [1], [2]. The objective evaluation of human taste sensation using electrophysiological signals can facilitate early diagnosis of taste diseases. Participants’ responses are not limited to verbal descriptions compared to traditional subjective evaluation methods. As a result, bias in assessing taste function is substantially reduced [3]. Electroencephalography (EEG) is one of the most common physiological measurements to characterize human taste sensation [4]–[6]. In addition, many studies have confirmed that taste sensation induces facial muscle activity, which can be measured through electromyography (EMG) [7]–[9].