Integration of Transfer Learning and Self-Attention for Spontaneous Micro-Expression Recognition

Micro-expressions are involuntary facial expressions which cannot hide true mental state of a person. These expressions appear for a very short duration of time and thus are very difficult to spot and recognize. Performance of deep learning models becomes stagnant due to overfitting; transfer learning helps to address this issue. Although, spotting and classification of micro-expression is a challenging task, but self-attention mechanism can easily detect relevant facial region required for micro-expression recognition. Considering these advantages of transfer learning and self-attention mechanism, in this work, two deep CNN models - Dual Stream ResNet18 (DSResNet) and Dual Stream ResNet18 Attention (DSResNetAtt) are proposed which exploits transfer learning through ResNet18 and self-attention for spontaneous Micro-expression recognition (MER). To demonstrate the effect of self-attention, the model is evaluated in terms of classification accuracy on three benchmark datasets CASME-I, CASME-II, and SAMM. Performance of the model has also been compared with state-of-the-art existing models and comparative analysis shows its efficiency in terms of classification accuracy.

I. Introduction

Facial expression is an important aspect of social interaction among human beings. Basic facial expressions include anger, contempt, disgust, fear, happy, sad, and surprise. Facial expressions can be broadly categorized as macro and micro-facial expressions. Macro expressions are visible, prolonged in nature and can be easily identified by human beings. However, these expressions can be suppressed, posed and disguised to hide true emotion of a person. Micro-expressions on the other hand, are involuntary in nature, cannot be posed, visible for a very short duration of time (0:04 sec to 0:50 sec) and reflect true emotion of a person. Based on these properties, micro-expression recognition (MER) has a variety of potential applications such as lie detection, security, surveillance systems, online learning, entertainment, health-care systems (depression detection, clinical diagnosis), and forensics. The acquisition of micro-expressions is difficult as compared to macro-expressions because they appear for a very short duration of time. However, with the advancement of data acquisition tools and software, research in the domain of MER has gained attention in the past few years.