Audio-Video Based Fusion Mechanism Using Deep Learning for Categorical Emotion Recognition

Multimodal emotion recognition is an advanced version of an artificial emotional intelligence system. Various solutions have been proposed to enhance the performance and emotion recognition rate. Multimodal is also one of the recent advanced research areas to classify and recognize emotions in realtime, consisting of physiological signals, facial expressions, audio-visual, audio-text, and video-text fusion. The proposed work uses advanced deep learning models to recognize the seven categorical emotions from the audio and video modality. Different modalities' feature extraction techniques and landmarks, such as Audio and Video, play a vital role in recognizing emotions. It also illustrates the Audio-video-based feature fusion mechanism. It computes the complementary relationship between the different modalities and shows the improved performance using the RESNET20 model for audio modality and the 3DCNN model for video modality on the RAVDESS dataset. Opensmile toolkit is used to extract the compelling features from the audio model and fuse them with extracted features of video modality. Feature fusion mechanism fuses the extracted features of audio and video data to classify and enhance the accuracy of recognized categorical emotions. It also briefly discusses the various audio-video features fusion approaches, computational accuracy and robustness of the proposed model. It shows a result of 71.3% and 69.7%. Accuracy on validation and test datasets outperforms the state-of-the-art models in the multimodal emotion recognition domain with a fusion mechanism.