Multimodal Speech Emotion Recognition Using Deep Learning and the Impact of Data Balancing

In recent years, many studies have investigated the potential uses of recognizing emotions from speech in a variety of sectors, attracting a lot of attention to this topic. The performance of Speech Emotion Recognition (SER) is significantly impacted by effective emotional feature extraction from speech. In this paper, we have presented a novel approach of multimodal SER using the interactive emotional dyadic motion capture database. Our primary focus lies in overcoming the common challenges of imbalanced dataset in emotion recognition research, which often leads to reduce the accuracy. To address this issue, we have employed a data balancing technique. Our multimodal model initially has shown promising results even before the implementation of DB techniques, but the incorporation of DB techniques marked the turning point in our study and has been able to significantly improve the model's performances such as, precision to 77.64 %, recall to 77.21 %, F1-score to 77.42 % and accuracy to 79.32%. These outcomes unequivocally show the value of DB in overcoming the constraints of unbalanced dataset and support the efficacy of our proposed multimodal strategy for SER, along with providing insightful information for prospective uses in real-world situations as well as future developments in this field.