Dual Flow Framework on Bimodality Emotion Recognition Based on Facial Expression and Eye Movement

Emotion affects many aspects of our daily life. Lack of emotional communication in online learning results in poor learning effect and learning experience. This paper discusses the feasibility of recognizing emotions by using eye movements and facial expressions data together. A framework for identifying the learner’s emotional state based on expression and eye movement bimodality recognition is proposed based on the complementarity of eye movement and facial expression data. The video stream sequence is used to obtain the learner’s facial expression and eye movement information as dual-channel data, which is input into the data stream framework, and the machine learning method is used to predict the learner’s emotional state. Dual-channel data flow framework proposed in this paper can be used not only to discover learners’ emotional states in learning environments, but also can be applied to the area of identifying mental disorder and human-computer interaction.