M$^{3}$SA: Multimodal Sentiment Analysis Based on Multi-Scale Feature Extraction and Multi-Task Learning

Sentiment analysis plays an indispensable part in human-computer interaction. Multimodal sentiment analysis can overcome the shortcomings of unimodal sentiment analysis by fusing multimodal data. However, how to extracte improved feature representations and how to execute effective modality fusion are two crucial problems in multimodal sentiment analysis. Traditional work uses simple sub-models for feature extraction, and they ignore features of different scales and fuse different modalities of data equally, making it easier to incorporate extraneous information and affect analysis accuracy. In this paper, we propose a Multimodal Sentiment Analysis model based on Multi-scale feature extraction and Multi-task learning (M$^{3}$SA). First, we propose a multi-scale feature extraction method that models the outputs of different hidden layers with the method of channel attention. Second, a multimodal fusion strategy based on the key modality is proposed, which utilizes the attention mechanism to raise the proportion of the key modality and mines the relationship between the key modality and other modalities. Finally, we use the multi-task learning approach to train the proposed model, ensuring that the model can learn better feature representations. Experimental results on two publicly available multimodal sentiment analysis datasets demonstrate that the proposed method is effective and that the proposed model outperforms baselines.

I. Introduction

The sentiment is essentially important in human decision-making, social activities, and creativity. Sentiment analysis is an indispensable part of human-computer interaction [1]. The earliest systematic research in sentiment analysis with machine learning dates back to 2002. Pang et al. [2] utilized three machine learning algorithms to analyze the sentiment of movie reviews and verified the feasibility of employing computers to analyze human sentiment. Subsequently, many machine learning algorithms, such as Naive Bayes [3] and support vector machine [4], had been exploited for sentiment analysis. Human sentiments are generally composed of multiple modalities, and different modalities may express opposite sentiments, which results in less accuracy if we analyze the sentiment with only unimodal data.