Multimodal Video Emotional Analysis of Time Features Alignment and Information Auxiliary Learning

With the rapid growth of various images and video data on the network platform, multimodal emotional analysis and recognition have become an increasingly popular research field. Inspired by the human emotional judgment and cross information between different characteristics, this paper proposed a multimodal video emotional analysis network of time features alignment and information auxiliary learning(METI). This network not only fully consider the time alignment between modals to obtain the preliminary information but also pay attention to the auxiliary information in the video and make emotional judgments. METI network is mainly composed of the following three parts: Time alignment network pays attention to time alignment information between different modes; Prepose feedback module uses the auxiliary information in the video to simulate the emotional judgment process; Multimodal gate control module pays attention to the emotional correlation of the context. The comparison of experimental results shows that the performance of the METI model on the dataset exceeds the current most advanced emotional analysis model. At the same time, it has been proved from ablation experiments that the modules and methods proposed in this paper can improve the accuracy of emotion recognition by nearly 4%, and has been respectively proved in the seven classification experiment of emotions and three classification experiment of sentiment.