BEmoFusionNet: A Deep Learning Approach For Multimodal Emotion Classification in Bangla Social Media Posts

Multimodal emotion classification, incorporating both image and text modalities, has gained significant attention due to the exponential growth of multimedia data. This research aims to develop a robust system for multimodal emotion classification in Bangla social media content, utilizing both image and text data. Despite limited resources, understanding emotions in Bangla is crucial for mental health interventions. Unlike previous studies focused on well-resourced languages, this research addresses the gap by specifically targeting regional languages like Bangla, going beyond the traditional positive, negative, and neutral classes. This work presents a multimodal Bangla social post dataset containing 4660 samples. Transfer learning techniques, utilizing pre-trained models like ResNet50, VGG16, and InceptionV3, extract visual features, while deep learning architectures such as BiLSTM and CNN are employed for textual content analysis. Multimodal learning techniques, including feature fusion and decision fusion, are explored to combine visual and textual representations. We evaluated the feature fusion of InceptionV3 and BiLSTM features on our Bangla social media post dataset. Our approach achieved a weighted f1-score of 77.50%.