pix2gestalt: Amodal Segmentation by Synthesizing Wholes

We introduce pix2gestalt, a framework for zero-shot amodal segmentation, which learns to estimate the shape and appearance of whole objects that are only partially visible behind occlusions. By capitalizing on large-scale diffusion models and transferring their representations to this task, we learn a conditional diffusion model for reconstructing whole objects in challenging zero-shot cases, including examples that break natural and physical priors, such as art. As training data, we use a synthetically curated dataset containing occluded objects paired with their whole counterparts. Experiments show that our approach outperforms supervised baselines on established benchmarks. Our model can furthermore be used to significantly improve the performance of existing object recognition and 3D reconstruction methods in the presence of occlusions.

1. Introduction

Although only parts of the objects in Figure 1 are visible, you are able to visualize the whole object, recognize the category, and imagine its 3D geometry. Amodal completion is the task of predicting the whole shape and appearance of objects that are not fully visible, and this ability is crucial for many downstream applications in vision, graphics, and robotics. Learned by children from an early age [30], the ability can be partly explained by experience, but we seem to be able to generalize to challenging situations that break natural priors and physical constraints with ease. In fact, we can imagine the appearance of objects during occlusions that cannot exist in the physical world, such as the horse in Magritte's The Blank Signature.