Multimodal Emotion Recognition Based on Feature Fusion

In the field of human-computer interaction, human emotion recognition is a challenging problem, and it is also a key link to achieve barrier-free communication between human and machine. At present, most of the emotion recognition algorithms are constructed based on single modal social information, and the recognition results are one-sided and easily disturbed. The recognition accuracy is often difficult to meet the practical requirements after being separated from specific social environment conditions. Based on the above situation and problems, this paper adopts multimodal input and simultaneously includes three modal information of audio, text and facial expression to recognition emotion. Three single modal emotion recognition models are proposed based on three different input information, and the multimodal emotion recognition model are constructed by different feature fusion methods. The experimental results showed that the accuracy of multimodal model on the CH-SIMS dataset was 93.92%. In addition, compared with other emotion recognition models, the effectiveness of the proposed method is verified.

I. Introduction

With the development and wide application of artificial intelligence technology, it has brought great convenience and fun to human life. The fluency and harmony of human-computer interaction has become the current research goal, but emotion recognition is an important part of human-computer interaction [1]. Due to the popularity of a large number of emotion dataset available in various forms, it provides a certain support for further improving the ability of emotion recognition. How to achieve more accuracy analysis and judgement for the subject’s emotional state, is the common pursuit of researchers in current.