Sparsity-based joint gaze correction and face beautification for conferencing video

A well-known problem in video conferencing is gaze mismatch. Instead of relying exclusively on online captured data for rendering, a recent work first trains offline dictionaries using a large image database of movie and TV stars to learn "beautiful" features. During real-time conferencing, one can then simultaneously correct gaze and beautify the subject's facial components in single images by seeking sparse linear combination of pre-trained dictionary atoms for face reconstruction. Extending on this work, we focus on joint gaze correction / face beautification for video. First, we define a large search space invariant to scale, shift and rotation for facial feature beautification based on SIFT. We then address two practical issues unique to video: i) how beautified results can be temporally consistent across group of pictures (GOP), and ii) how blinking eyes can be beautified even though the training database contains only open-eye facial images. Experimental results show that our method achieves the desired temporal consistency, and the blinking process is smooth and natural.

1. Introduction

Video conferencing is now widely used across the globe via available tools like Skype, Google Hangout, etc. However, the user experience enabled by existing tools is still inferior to live face-to-face communication. One of the glaring problems is gaze mismatch [1]: a capturing web camera is typically located above or below a display monitor, while the conference subject looks at his counterpart rendered at the screen center. This means that the participant's gaze direction is not aligned with the camera's line-of-sight, and thus the parties cannot converse eye-to-eye. See Fig. 1(a) for an illustration.