Orthogonal Synthesis: Data Augmentation for Multimodal Sentiment Analysis

Multimodal sentiment analysis (MSA) is an emerging research field that identifies or quantifies opinions in the form of available textual, acoustic, and visual modalities. The state-of-the-art (SOTA) MSA highly relies on the pre-trained and transformer-based model like BERT that provides contextual representations of the textual modality. However, how to exploit the rich information from the non-textual modalities is undermined yet. These non-textual modalities consist of not only sentiment-related features but also unwanted confounding features, which are difficult to handle since there is no standard preprocessing, like word embedding, and no appropriate corpora of the non-textual modalities for pre-training. In this paper, we propose a data augmentation approach, Orthogonal Synthesis (OrthSyn), to alleviate the side effect of the confounding feature, enabling neural networks to focus on the sentiment-related feature. To obtain augmented data whose confounding features are less distinctive than the original data, we synthesize (Syn) samples with gradient ascent on an identity classification task. To keep the sentiment-related features intact in the synthesized data, we orthogonalize (Orth) the gradients during network optimizations in the identity classification task and the sentiment analysis task. Our experiments evaluate the OrthSyn and its variants on the commonly used CMU-MOSI and CMU-MOSEI datasets. The results show that OrthSyn outperforms SOTA methods. We further investigate the detailed effect of OrthSyn and visualize how the OrthSyn improves performance as the training proceeds.