Multimodal Sentiment Analysis under modality deficiency with prototype-Augmentation in software engineering

Sentiment analysis has a wide range of promising applications in software engineering, and the development of deep learning has demonstrated that the uniform representation of different modalities can improve the model performance of sentiment analysis. However, in practical applications, multimodal sentiment analysis always faces unsatisfactory situations, especially when the modality has missing samples, most models may fail. For example, social dynamics of technicians in developer communities can face modality unavailability due to privacy settings. Several existing works based on deep learning and regularization methods have explored the modal missing problem, but these works cannot balance the cases of modal general missing (rate < 50%) and severe missing (rate ≥ 50%), and do not consider the resource consumption during model inference. Therefore, in this paper, we proposed a prototype augmented multimodal teacher-student network (PAMD) to address the above issues. Specifically, a multi-level and multi-origin distillation strategy is used to minimize the required resources and inference time, and prototype augmentation is used to guarantee the performance of the model when a modality is severely missing. Extensive experiments are conducted on different benchmark datasets to explore a network that balances performance and resource consumption. And It achieves good results in different modalities of missing cases.

I. Introduction

Multimodal Sentiment Analysis (MSA) is widely used in all areas of software engineering and is used throughout the software engineering lifecycle. For example, as one of the most important parts of the software engineering lifecycle, software development is a highly collaborative activity that is highly influenced by the emotional state of developers. Negative affective states can make developers less productive in software projects and can easily introduce software defects [33], while positive affect is expected to increase developer productivity [1]. In addition, obtaining valid sentiment status from APP reviews, technical Q&A sites such as Stack Overflow, and developers' comments on APIs is crucial for subsequent product improvement and service refinement. For example, the comment "I am not able to deploy my App Engine project locally." on the Java API predicts a negative sentiment, which in turn leads to the possibility of optimizing the mentioned "App Engine" service. This shows that it is necessary and essential to conduct research on sentiment analysis in software engineering.