Few-Shot Physically-Aware Articulated Mesh Generation via Hierarchical Deformation

We study the problem of few-shot physically-aware articulated mesh generation. By observing an articulated object dataset containing only a few examples, we wish to learn a model that can generate diverse meshes with high visual fidelity and physical validity. Previous mesh generative models either have difficulties in depicting a diverse data space from only a few examples or fail to ensure physical validity of their samples. Regarding the above challenges, we propose two key innovations, including 1) a hierarchical mesh deformation-based generative model based upon the divide-and-conquer philosophy to alleviate the few-shot challenge by borrowing transferrable deformation patterns from large scale rigid meshes and 2) a physics-aware deformation correction scheme to encourage physically plausible generations. We conduct extensive experiments on 6 articulated categories to demonstrate the superiority of our method in generating articulated meshes with better diversity, higher visual fidelity, and better physical validity over previous methods in the few-shot setting. Further, we validate solid contributions of our two innovations in the ablation study. Project page with code is available at meowuu7.github.io/few-arti-obj-gen.

1. Introduction

Generative models have aroused a wide spectrum of interests in recent years for their creativity and broad down-stream application scenarios [29], [30], [34], [17], [8], [26]. Specific to 3D generation, a variety of techniques such as denoising diffusion [23], [42], [6], [39] have also been discussed for a while. Among them, mesh generation is indeed important since the mesh representation can support a wider range of downstream applications such as rendering and physical simulation compared to other representations such as point clouds. Existing works mainly focus on generating meshes for whole objects [8], [26], [6], [19], [30] considering without modeling object functionalities at all. Besides, they mainly rely on reconstructing meshes from other kinds of representations such as implicit fields [8], [6], [19] instead of generating meshes directly. In this work, we go one step further and consider mesh generation for articulated objects that can support physically realistic articulations. This not only helps understand the object distribution in real-world assets, but also allows an intelligent agent to learn segmenting [20], [22], tracking [36], reasoning [10] and manipulating [38] articulated objects through a simulation environment. We focus on the articulated mesh generative model that can generate object meshes with diverse geometry, high visual fidelity, and correct physics.