Improving Speech Emotion Recognition through Hierarchical Classification and Text Integration for Enhanced Emotional Analysis and Contextual Understanding

Speech emotion recognition (SER) systems are designed to classify spoken audio samples into different emotion categories. However, misclassifying emotional samples and predicting them as neutral remains a challenging problem in these systems. One primary contributing factor to this issue is the limitation of speech features to recognize emotions from neutral spoken samples that convey emotional context, as these features do not account for contextual or meaning-based features. To address this issue and improve the recognition performance in SER, we propose a hierarchically structured classification model and integrate text features as a supportive modality to address the misclassification of emotional samples. Text-based features provide valuable contextual information that can aid in identifying emotional content in otherwise neutral speech. This work could be potentially applied in various fields, such as healthcare, education, and entertainment, where recognizing emotions from speech can be crucial for effective communication and decision-making.