Ensemble Integration of Deep Learning Models for Gender-Based Speech Emotion Recognition

Speech emotion recognition (SER) holds significant importance in numerous applications, as it heavily relies on the precise extraction of acoustic information from speech signals. However, recognizing emotions within a complete utterance often poses challenges due to inherent ambiguity and varying probabilities assosiated with different emotions. Previous approaches have failed to account the acoustic variances between male and female speakers by employing combined dataset. Furthermore, relying on a single model can lead to limitations such as reduced robustness, overfitting, and limited generalization. To mitigate these issues, this paper proposes an ensemble model that combines Convolutional Neural Network (CNN), Residual CNN (RCNN) and Long Short-Term Memory (LSTM) architectures and utilizes features, extracted using Mel Frequency Cepstral Coefficients (MFCC) on a gender-differentiated dataset that provides comprehensive and robust approach to speech emotion recognition. This proposed ensemble model is trained and evaluated on RAVDESS, CREMA-D, SAVEE and TESS datasets, resulting in weighted mean accuracy of 95.54%. The obtained results clearly indicate that the utilization of the Ensemble model instead of individual models improved the accuracy. The Ensemble model effectively encompasses diverse aspects of the data by leveraging the unique strengths of each model.