Recognition and mapping of facial expressions to avatar by embedded photo reflective sensors in head mounted display

We propose a facial expression mapping technology between virtual avatars and Head-Mounted Display (HMD) users. HMD allow people to enjoy an immersive Virtual Reality (VR) experience. A virtual avatar can be a representative of the user in the virtual environment. However, the synchronization of the the virtual avatar's expressions with those of the HMD user is limited. The major problem of wearing an HMD is that a large portion of the user's face is occluded, making facial recognition difficult in an HMD-based virtual environment. To overcome this problem, we propose a facial expression mapping technology using retro-reflective photoelectric sensors. The sensors attached inside the HMD measures the distance between the sensors and the user's face. The distance values of five basic facial expressions (Neutral, Happy, Angry, Surprised, and Sad) are used for training the neural network to estimate the facial expression of a user. We achieved an overall accuracy of 88% in recognizing the facial expressions. Our system can also reproduce facial expression change in real-time through an existing avatar using regression. Consequently, our system enables estimation and reconstruction of facial expressions that correspond to the user's emotional changes.

1 Introduction

Facial expression is one of the most important features of nonverbal expressions in human communication. In a virtual environment’ it is possible to synchronize the facial expression of the virtual avatar with the user's expression using facial capturing technology. However, in case of a head-mounted display (HMD), capturing facial expressions is a challenging due to optical occlusions. In order to synchronize the expression of the avatar with the user's expression in HMD applications, there needs to be a system that can estimate facial expressions while the user is wearing an HMD. Previous researches often extracted facial features from image sequences for recognizing facial expression. This kind of camera-based recognition is not suited for an HMD-based system because the HMD would cover the user's face and restrict his/her body movements.