Speech Emotion Recognition based on Multi COATTENTION Acoustic Feature Fusion

Speech Emotion Recognition (SER) is a technology aimed at extracting emotional information from speech signals, thereby discerning the emotional state of the speaker. Its objective is to equip machines with empathetic capabilities akin to those of humans. The ability to extract high-quality, deep, and comprehensive speech representations from audio often determines the robustness of emotion analysis. In this paper, we focus on modeling speech using spectrograms, MFCCs (Mel-frequency cepstral coefficients), and raw waveforms. We employ CNNs (Convolutional Neural Networks), BiGRUs (Bidirectional Gated Recurrent Units), and self-supervised Wav2Vec-based models [1] to extract deep acoustic features, each applied separately. Taking into account the complementarity among different acoustic features, we introduce a Multi CO-ATTENTION Mechanism to enhance the acoustic characteristics. Additionally, feature fusion is performed at the final stage of the model. We evaluated our proposed method on the IEMOCAP dataset and conducted ablation studies, achieving promising results.