Say CHEESE: Common Human Emotional Expression Set Encoder and Its Application to Analyze Deceptive Communication

In this paper we introduce the Common Human Emotional Expression Set Encoder (CHEESE) framework for objectively determining which, if any, subsets of the facial action units associated with smiling are well represented by a small finite set of clusters according to an information theoretic metric. Smile-related AUs (6,7,10,12,14) in over 1.3M frames of facial expressions from 151 pairs of individuals playing a communication game involving deception were analyzed with CHEESE. The combination of AU6 (cheek raiser) and AU12 (lip corner puller) are shown to cluster well into five different types of expression. Liars showed high intensity AU6 and AU12 more often compared to honest speakers. Additionally, interrogators were found to express a higher frequency of low intensity AU6 with high intensity AU12 (i.e. polite smiles) when they were being lied to, suggesting that deception analysis should be done in consideration of both the message sender's and the receiver's facial expressions.

I. Introduction

Of the witnesses shown in Fig. 1 (W1-W4), can you guess who is being honest and who is lying? What about the interrogators (15 −18) — can you tell which of them are being lied to and which are being told the truth? Could their smiles be giving us any clues?