3D Face Shape Regression From 2D Videos with Multi-Reconstruction and Mesh Retrieval

This paper introduces our submission to the 2nd 3DFAW Challenge. To get a high-accuracy 3D dense face shape based on 2D videos or multiple images, a framework which is consist of multi-reconstruction branches and a mesh retrieval module, is proposed to effectively utilize the information of all frames and the results predicted by all branches. The recent state-of-the-art methods based on single-view and multi-view are introduced to form an ensemble of independent regression networks. The candidate 3D shape of each branch is synthesized by weighted linear combination of the results on all frames to boost the depth estimation and invisible regions reconstruction. Finally, the best fitting mesh is retrieved according to the distance between the synthesized texture and the ground truth texture. Experiment results show that our approach obtains competitive results near the accuracy of "pseudo" ground truths, and achieves superior performance over most of submissions by other teams in the testing phases.

1. Introduction

Face alignment is critical to face analysis applications, such as face identification [2], [29], face tracking [25], and face synthesis [8], [6], [13]. Compared with 2D face alignment methods [24], [18], [14], [23], 3D face alignment is more robust to variation of occlusions and out-of-plane rotations, and has stronger representational power for describing face shapes [20], [27], [5], [19]. 3D face alignment and reconstruction have made rapid advances in recent years, especially after the utilization of deep convolution neural networks (CNN) for solving the problem [5], [1], [27], [7], [21]. Previous approaches can be divided into two categories according to input data type, the methods based on single-view and the methods based on multi-view. Due to space constrains, we mainly focus on the recently proposed methods related with our work from the above two categories, then discuss their advantages and drawbacks, respectively.