Implicit Emotion Relationship Mining Based on Optimal and Majority Synthesis From Multimodal Data Prediction

Emotion is precious, useful for many applications such as public opinion detection and psychological disease prediction. Emotion recognition on multimodal data has attracted extensive attention. Although modifying model structure or multimodal feature fusion methods have contributed a lot to emotion recognition, little attention is paid to mining implicit emotion relationship. In this article, implicit emotion relationship consists of emotion distribution, confusion, and transfer. Emotion distribution allows multiple emotions in one sample, while confusion and transfer imply the prediction confusion and bias. In order to mine implicit emotion relationship in multimodal data, this article employs three image and two text classification models to recognize emotions, respectively. Two prediction emotion synthesis methods (optimal prediction emotion synthesis and majority prediction emotion synthesis) are proposed to synthesize the outputs of multiple models. Based on the results of two emotion synthesis methods, emotion distribution on samples is obtained. Emotion confusion and transfer among different emotion samples are analyzed by relative entropy and Jensen–Shannon divergence. Implicit emotion relationship mining has potential not only in the interpretation of model performance, but also in guiding the development of emotion recognition as prior knowledge. Finally, we take topic scenario as an instance to mine implicit emotion relationships.

EMOTION IS PRECIOUS, useful for many applications such as public emotion detection and psychological disease prediction. Emotion mining has attracted extensive attention. Many scholars employ multimodal information fusion to mine the potential relationship between low-dimensional data features and subjective emotions, which have achieved good results. With the development of machine learning technology such as deep learning, part of emotion recognition problems are solved, but there are still many challenges.a)

Emotion confusion is the main reasons for an emotion recognition error. Emotion confusion means that the emotion expressed in samples is often a mixture of multiple emotions instead of being independent.

b)

The methods of multimodal feature fusion lack theoretical basis and guidance of emotion relationship in multimodal data. Multimodal data fusion mainly includes three approaches: early fusion, late-fusion, and intermediate-fusion. How to fuse the multimodal data and define the fusion weights of different modal data is also a problem that needs to be considered.

c)

Large-scale datasets with high quality for emotion recognition are very rare. In classification tasks with absolutely correct labels, manually labeled datasets are usually of high quality. However, emotion is subjective. People with different cultural backgrounds and social experiences may carry different emotions on the same text or image. Multimodal datasets for emotion recognition need a large number of people to label them. This will reduce the recognition error caused by human bias as much as possible, but it also means the high cost of data collection and annotation.