Tensor Correlation Fusion for Multimodal Physiological Signal Emotion Recognition

As an essential challenge within the realm of affective computing, emotion recognition assumes a vital role in bestowing computers with a higher level and comprehensive intelligence. Furthermore, it has emerged as a crucial research topic in both human–computer interaction (HCI) and medical rehabilitation related to mental illnesses. However, the related fusion studies for modeling physiological signals in emotion recognition are less based on multimodal coordinated representation and lack the exploration of multimodal physiological signal correlation. In this article, we propose a tensor correlation fusion framework for emotion recognition based on multimodal physiological signals. After extracting effective features from various physiological signals, the coordinated representation module of the framework first simultaneously learns the linear correlation of all input physiological signals based on the covariance tensor. An optimized solution strategy is constructed to obtain the coordinated representation corresponding to each physiological signal. Finally, an emotion recognition module fuses the correlation information of the coordinated representation of different physiological signals as input to the emotion recognition classifier. This framework constructs coordinated representation by introducing a strategy to simultaneously capture the correlation among multiple physiological signals, providing a fresh perspective with a well-defined mathematical foundation for the fusion of multimodal physiological signals in the realm of emotion recognition. The experiments conducted on the DEAP dataset demonstrate that compared with related methods, the framework achieves relatively higher emotion recognition performance while obtaining a coordinated representation of multimodal physiological signal correlations of emotions, all while achieving superior processing speed.