Action unit selective feature maps in deep networks for facial expression recognition

Facial expression recognizers based on handcrafted features have achieved satisfactory performance on many databases. Recently, deep neural networks, e. g. deep convolutional neural networks (CNNs) have been shown to boost performance on vision tasks. However, the mechanisms exploited by CNNs are not well established. In this paper, we establish the existence and utility of feature maps selective to action units in a deep CNN trained by transfer learning. We transfer a network pre-trained on the Image-Net dataset to the facial expression recognition task using the Karolinska Directed Emotional Faces (KDEF), Radboud Faces Database(RaFD) and extended Cohn-Kanade (CK+) database. We demonstrate that higher convolutional layers of the deep CNN trained on generic images are selective to facial action units. We also show that feature selection is critical in achieving robustness, with action unit selective feature maps being more critical in the facial expression recognition task. These results support the hypothesis that both human and deeply learned CNNs use similar mechanisms for recognizing facial expressions.

I. Introduction

Facial expressions are one of the most significant manifestations of human emotions. Facial expression recognition is gaining increased attention, as it can provide an additional input to human-computer interfaces. Approaches can be categorized as geometry-based, Action Unit (AU)-based or appearance-based. Most previously proposed methods rely upon extracting hand-crafted features. Geometry-based approaches extract features by tracking facial landmark points and modeling the geometrical relationship between them [1]. AU-based methods are grounded in the Facial Action Coding System (FACS) proposed by Paul Ekman et al. [2], which is widely utilized to encode facial expressions according to the localized muscle movements of facial regions, called Action Units. These approaches train individual AU detectors first and then analyze the combinations of AUs to classify facial expression based on the FACS [3]–[5]. Appearance-based methods usually use texture features extracted from local image patches, such as local binary pattern (LBP) [6], Gabor [7], [8], Haar [9] or scale invariant feature transform (SIFT) [10] features. In all of the above approaches, extracted features are passed to a classifier, e.g. a support vector machine, neural network, or Bayesian classifier, for the final recognition. They have achieved good accuracy on a number of databases, such as the extended Cohn-Kanade (CK+) database [11], the Japanese Female Facial Expression (JAFFE) database [12], the MMI database [13], etc.