Learning a Bimodal Emotion Recognition System Based on Small Amount of Speech Data

This paper presents a bimodal emotion recognition system based on the voice and text information using small amount of speech data. Specifically, speech is divided into voice and text to learn emotion classifiers for each modal. The probabilities obtained from these emotion classifiers are weighted based on Mehrabian’s rule and summed up for each emotion to calculate the final score for bimodal emotion recognition. To create a highly accurate system while solving the problem that there are few Japanese speech data with emotion labels, we propose a novel data augmentation method and employ a transfer learning approach based on a pre-trained VGG16 model and a fine-tuned Bidirectional Encoder Representations from Transformers (BERT) model on the tweets. In order to prove the effectiveness of the proposed method, we revealed the recognition results for 7 emotional states, i.e., anger, sadness, joy, fear, surprise, disgust, and neutral. The experiment result suggested that our novel data augmentation method improved the accuracy and that bimodal predictions based on voice and text-based outperformed the single-model predictions.