Research on Multimodal Intelligence with Feature-level Architecture for Affective Computing

From unimodal to multimodal has been increasingly challenging for Affective Computing (AC), the main difficulty of Multi-modal recognition (MER)lies in the alignment of diverse features that lead to reduce the accuracy of affective classification, we present a feature-level architecture for Multimodal emotion classification. Using BERT and BiGRU to extract text features, training Viterbi algorithm via Hidden Markov Model (HMM) model to generate audio features, then representing image features as three-dimensional vector with Convolutional neural network (CNN), meanwhile, conducting Gaussian Diffusion Process (GDP) to achieve data synthetic on Electroencephalography (EEG). Finally exploring a Multi-modal task algorithm to produce classification on multimodal feature vectors. Experiments show that multimodal emotion classification can be effectively performed, and that the joint use of facial expression, body-gesture, speech and physiological signals modalities can lead to error rate reductions of up to 5.26% as compared to the best performing single modality.