LeGO: Leveraging a Surface Deformation Network for Animatable Stylized Face Generation with One Example

Recent advances in 3D face stylization have made significant strides in few to zero-shot settings. However, the degree of stylization achieved by existing methods is often not sufficient for practical applications because they are mostly based on statistical 3D Morphable Models (3DMM) with limited variations. To this end, we propose a method that can produce a highly stylized 3D face model with desired topology. Our methods train a surface deformation network with 3DMM and translate its domain to the target style using a paired exemplar. The network achieves stylization of the 3D face mesh by mimicking the style of the target using a differentiable renderer and directional CLIP losses. Additionally, during the inference process, we utilize a Mesh Agnostic Encoder (MAGE) that takes deformation target, a mesh of diverse topologies as input to the stylization process and encodes its shape into our latent space. The resulting stylized face model can be animated by commonly used 3DMM blend shapes. A set of quantitative and qualitative evaluations demonstrate that our method can produce highly stylized face meshes according to a given style and output them in a desired topology. We also demonstrate example applications of our method including image-based stylized avatar generation, linear interpolation of geometric styles, and facial animation of stylized avatars.

1. Introduction

Crafting animatable stylized 3D avatars that encapsulate both personal identity and character sty le requires extensive efforts from skilled artists. When creating animated films, the artists design stylized 3D avatars whose facial appearance matches the theme of the entire content while putting careful effort into preserving the idiosyncrasy of the actors. Similarly, on social media, artists create numerous stylized presets so that the combinations of these presets can represent diverse identities. Figure 1.

(a) The proposed method demonstrates robustness to unseen face identities and topologies and effectively generates stylized output faces with desired topologies. (b) Our stylized avatars can be animated using 3DMM blend shapes.