An Improved Method to Recognize Hand-over-Face Gesture based Facial Emotion using Convolutional Neural Network

An occlusion on face is typically considered as a major obstacle in recognizing emotion through facial expression. However, the hand-over-face occlusion provides useful information to recognize emotion precisely. The existing emotion recognition methods with or without hand gesture identify basic emotions such as happy, sad, angry, surprise, neutral, fear and disgust. However, additional emotions such as confident, making decision, scared, ashamed, angry and ok sign need to be identified for modern applications. In this paper, we present an empirical evaluation of an improved Hand-over-Face Gesture based Facial Emotion Recognition Method (HFG_FERM) that identifies additional unexplored emotions along with basic emotions. The proposed method includes extensive coding schema with additional hand signs to identify unexplored emotions. Additionally, it integrates convolution neural network in its design for deep and automatic feature extraction as well as for classification of emotions. The performance of HFG_FERM is evaluated under four distinct scenarios. The experimental results show that the HFG_FERM considerably increases emotion recognition accuracy.