Face Frontalization for Cross-Pose Facial Expression Recognition

In this paper, we have explored the effect of pose normalization for cross-pose facial expression recognition. We have first presented an expression preserving face frontalization method. After face frontalization step, for facial expression representation and classification, we have employed both a traditional approach, by using hand-crafted features, namely local binary patterns, in combination with support vector machine classification and a relatively more recent approach based on convolutional neural networks. To evaluate the impact of face frontalization on facial expression recognition performance, we have conducted cross-pose, subject-independent expression recognition experiments using the BU3DFE database. Experimental results show that pose normalization improves the performance for cross-pose facial expression recognition. Especially, when local binary patterns in combination with support vector machine classifier is used, since this facial expression representation and classification does not handle pose variations, the obtained performance increase is significant. Convolutional neural networks-based approach is found to be more successful handling pose variations, when it is fine-tuned on a dataset that contains face images with varying pose angles. Its performance is further enhanced by benefiting from face frontalization.

I. Introduction

Facial expression plays a significant role in human-human and human-machine interactions. Many real world applications require automatic facial expression recognition (FER) under unconstrained environments. Hence, facial images appearing in various poses should be handled by the developed FER systems. One way of addressing this problem is to perform face frontalization to eliminate the variations in facial appearance due to different views. However, it is critical to preserve facial expression, while performing pose normalization.