Multimodal Embeddings In Emotion Recognition Research

In recent years, the analysis and recognition of emotions has become an active research topic applied in various fields, including psychology. Methods of express personality diagnostics are of particular importance, as they correspond to the peculiarities of modern society with its fast pace of life and the need for rapid collection of reliable data. This article reviews common methods and algorithms for preprocessing input data to better understand emotion recognition problems. These include face detection in the input image, image smoothing method, principal component analysis (PCA), local binary patterns (LBP), optical flow (OF), Gabor filter, and others. The main types of classifiers used in the final stage of model building are considered: traditional classifiers, such as Support Vector Machines (SVA), and different types of neural networks (CNN, LSTM, RNN). The paper takes a multimodal approach based on three input modalities: Body posture, tone (speech), and facial expression. Each modality is represented by an independent vector extracted from an independently pre-trained (SSL) model. Given the multidimensional nature of the features extracted from SSL, we propose the use of a transformer-type neural network based on the attentional mechanism that can combine the multimodal features of SSL and achieve state-of-the-art results for the task of multimodal emotion recognition. Further efforts should be directed towards multimodal systems that are robust enough to adapt to the real world. This review is intended not only to serve as a reference for future research in emotion recognition, but also to provide potential readers with an overview of the work that has been done on this topic.