Multimodal Enhanced Sentiment Analysis with Hierarchical Feature Extraction

Multi-modal sentiment analysis (MSA), as a hot research topic in the fields of natural language processing, affective computing, and multimedia analysis, has gradually become an important method for sentiment analysis in social networks, especially in combining text and image modalities for data analysis. However, existing studies suffer from the issue of ineffective fusion of multimodal sentiment features. In this paper, we propose a method for multimodal enhanced sentiment analysis using hierarchical feature extraction (ME-SAHFE). When extracting text features using RoBERTa, our method further extracts features from each layer output by employing a Convolutional Neural Network (CNN) to enhance the representation capability of text features. Subsequently, the extracted text features are concatenated with image features and inputted into a multilayer perceptron (MLP) for effective fusion. Our method achieves significant improvements in accuracy, precision, recall, and F1 score in sentiment classification on the MVSA-Multi dataset.