Real-Time Facial Emotion Detection System Using Multimodal Fusion Deep Learning Architecture

One of the crucial applications in the area of computer vision is face emotion detection. Systems for detecting facial emotions are frequently utilized in fields of study such as the detection of human social and physiological interactions and the diagnosis of mental diseases. Identifying facial emotions with high accuracy is a challenging task. The capacity of the deep learning model to extract facial features from the input images determines how accurately emotions are detected. The deep learning model used in this work for emotion detection is the convolution and attention network (CoAtNet), which combines the convolutional neural networks (CNNs) with the vision transformers (ViTs). The FER2013 dataset is the one used for model training and validation. This work also includes training on various CNN models such as VGG16, ResNet50, DenseNet121, and Xception. Among all the models, the CoAtNet model provided the highest validation accuracy of 80.314%. To improve accuracy, the facial landmarks of the images are extracted and incorporated into the deep learning network through the Multimodal Fusion approach. By incorporating the facial landmarks through multimodal fusion, the validation accuracies of the deep learning networks are improved, and the highest accuracy of 83.458% is obtained with the CoAtN et model. Using the CoAtNet model, a face emotion detection system is implemented on the Raspberry Pi hardware, which detects the emotion of a person by capturing real-time images.