LaWNet: Audio-Visual Emotion Recognition by Listening and Watching

Audio-Visual Emotion Recognition (AVER) represents an emerging and highly innovative research domain that exploits the combined utilization of audio and visual cues to enhance the comprehension and interpretation of human emotions. In this study, we introduce a novel model for Audio-Visual Emotion Recognition, referred to as LaWNet, which effectively encodes and fuses the audio signal and the Mel spectrogram derived from the audio signal as unified audio features, circumventing the loss of latent features that typically occurs during audio feature extraction in previous methodologies. Furthermore, we propose a Random Sampling method that addresses the issues of inadequate robustness and suboptimal utilization of video data observed in prior techniques. Additionally, we devise a cross-attention based fusion module to facilitate the fusion of audio-visual features. Through comprehensive experiments, we substantiate the efficacy of our proposed methodology, as evidenced by the outstanding performance achieved by the LaWNet model on two widely employed datasets, namely RAVDESS and CREMA-D, attaining accuracy rates of 99.31% and 91.53% respectively. These remarkable results significantly surpass the performance of previous methods and establish a new state-of-the-art benchmark within the realm of Audio-Visual Emotion Recognition.