Image Ratio Features for Facial Expression Recognition Application

Video-based facial expression recognition is a challenging problem in computer vision and human-computer interaction. To target this problem, texture features have been extracted and widely used, because they can capture image intensity changes raised by skin deformation. However, existing texture features encounter problems with albedo and lighting variations. To solve both problems, we propose a new texture feature called image ratio features. Compared with previously proposed texture features, e.g., high gradient component features, image ratio features are more robust to albedo and lighting variations. In addition, to further improve facial expression recognition accuracy based on image ratio features, we combine image ratio features with facial animation parameters (FAPs), which describe the geometric motions of facial feature points. The performance evaluation is based on the Carnegie Mellon University Cohn-Kanade database, our own database, and the Japanese Female Facial Expression database. Experimental results show that the proposed image ratio feature is more robust to albedo and lighting variations, and the combination of image ratio features and FAPs outperforms each feature alone. In addition, we study asymmetric facial expressions based on our own facial expression database and demonstrate the superior performance of our combined expression recognition system.

I. Introduction

In the past decade, much progress has been made on face recognition [1]–[4]. However, computational facial expression analysis is still a challenging and attractive research topic in computer vision and intelligent human–computer interaction (HCI). Facial-expression-related research was launched by Ekman and Friesen [5] in the 1970s. In Ekman's early work, face actions were described in terms of the Facial Action Coding System (FACS) [7], and facial expressions were semantically coded with respect to seven basic but “universal” dimensions, i.e., neutral, anger, disgust, fear, joy, sadness, and surprise. However, in practice, automatic facial expression recognition by computer did not really begin until 1990s. Most approaches [9]–[17] are based on Ekman's theory for developing intelligent HCI at the current stage, although some researchers proposed approaches based on other emotion models, e.g., the valence/arousal dimensional model [8].