Multimodal Sentiment Analysis With Image-Text Correlation Modal

Multi-modality sentiment analysis in the Social Internet of Things (SIoT) is an emerging area of study. The integration of information from different modalities is the essence of multi-modal sentiment analysis tasks. In social networks, there is often a correspondence between the text and images within a comment. Users tend to use emotionally charged words to describe the subjects in the images. Therefore, exploring the interaction between text and images helps improve the accuracy of sentiment recognition. We propose an Image-Text Correlation(ITC) model for multimodal emotion analysis. ITC is a three-layered architecture consisting of a text layer, an image layer, and a fusion layer. The text layer employs a soft attention mechanism to transform word representations of the text section into sentence representations. In the image layer, we input the image into the VGG-16 network to obtain the image representation. In the fusion layer, we calculate the correlation between sentences and images. Based on this correlation, weights are assigned to the sentences, allowing us to construct new document vectors for classification. Experiments conducted on the Yelp dataset demonstrate the effectiveness of our modal for multimodal sentiment analysis. The dataset comprises comments with both images and text, labeled with ratings ranging from 1 to 5 stars (considered as five distinct sentiment categories).

I. Introduction

Sentiment analysis in the Social Internet of Things (IoT) holds various practical applications [1] [2]. It can be applied to analyze user feedback and sentiments regarding smart home systems, enabling continuous improvement and customization of automated settings. IoT devices in entertainment can use sentiment analysis to understand user preferences and emotional reactions to content, leading to personalized recommendations. With the development of the Internet, there is a large amount of multimodal data on social networks. Accordingly, we employ a multimodal approach for sentiment recognition.