Lightweight and Real-Time Framework for Facial Motion Retargeting

Facial action retargeting has been widely used in the field of film and games, but in order to improve accuracy, a large number of professional equipment is used for assistance, or it consumes huge resources for offline rendering, which is difficult to run on consumer-level equipment in real time. In this paper, we propose a framework that allows facial motion redirecting to run in real-time on devices with limited computing resources, only captured by a monocular RGB camera. We use multi-task learning to learn Identity Shape, expression parameters, and head pose at the same time, and use the Depthwise structure in MobileNet to reduce the amount of calculation and parameter size of the model. For the above three tasks, in addition to using the shared layer to extract the common features, the features of different granularities are extracted separately, and finally their loss functions are weighted and summed. Experiments on representative benchmark datasets demonstrate the effectiveness of our approach.