Enhanced Emotion Recognition Through Multimodal Fusion Using TriModal Fusion Graph Convolutional Networks

In the field of machine learning, self-supervised pre-training has made significant advancements, particularly in speech, vision, and natural language processing (NLP). This study introduces the TriModal Fusion Graph Convolutional Network (TFGCN), a novel framework that leverages the strengths of self-supervised pre-trained models to extract features across three modalities: text, audio, and visual, for emotion recognition.For text, we utilize GloVe, a method for generating rich word embeddings that provide nuanced contextual representations. In the audio domain, we employ Wav2Vec to extract speech features, while CLIP is used to capture intricate visual features.Our methodology focuses on the sophisticated use of cross-modal attention mechanisms, which refine the alignment and extraction of interactive information between the features of all three modalities. This cross-modal attention, combined with Graph Convolutional Networks (GCN), enables the fusion of multimodal information, capturing complex patterns and interactions crucial for accurate emotion recogni-tion.The proposed TFGCN architecture outperforms existing models, achieving a 2.92% absolute improvement in accuracy compared to the previous state-of-the-art on the benchmark IEMOCAP dataset. Additionally, we conduct experiments within each individual modality—text, audio, and visual—and compare these unimodal results with the multimodal approach, demonstrating the significant gains achieved through our multimodal fusion strategy.