Language Reinforced Superposition Multimodal Fusion for Sentiment Analysis

Nowadays, BERT has been effective in context extraction and widely utilized in multimodal fusion. However, many multimodal learning systems with BERT consider different modalities equally important, which ignores the huge capability difference between the pre-trained models and others. Moreover, equal multimodal fusion might introduce noise and limit the performance of these systems. To this end, we propose a superposition multimodal fusion framework to strengthen the importance of language modality with nonverbal information transmission. With the superposition strategy, language information will fuse with nonverbal information at different stages, which allows the model to fully learn language modality information. Besides, to ensure that language information is the main content of the fusion, we propose a comparison loss to help the reinforced cross-modal attention module better transfer nonverbal information to language features. Extensive experiments are conducted to compare with baselines and the results demonstrate that our method achieves the superior performance.

I. Introduction

Sentiment analysis has attracted lots of attention [1]–[4]. Previous researchers mainly focused on building models to analyze text data [5]. But not all text messages can reflect the speaker’s sentiment clearly. Sometimes we can not judge the speaker’s feelings from the sentence ‘I go to work by bus’. Nowadays, multimodal information has been widely used in research. ESE-FN [6] fuses multimodal features to deal with elderly activity recognition. Multimodal interaction framework is applied to study Image-text retrieval work [7]. In NLP fields, multimodal fusion strategies are utilized both in sentiment analysis and dialog system [8].