Amodal Completion via Progressive Mixed Context Diffusion

Our brain can effortlessly recognize objects even when partially hidden from view. Seeing the visible of the hidden is called amodal completion; however, this task remains a challenge for generative AI despite rapid progress. We propose to sidestep many of the difficulties of existing approaches, which typically involve a two-step process of predicting amodal masks and then generating pixels. Our method involves thinking outside the box, literally! We go outside the object bounding box to use its context to guide a pretrained diffusion inpainting model, and then progressively grow the occluded object and trim the extra background. We overcome two technical challenges: 1) how to be free of unwanted co-occurrence bias, which tends to regenerate similar occluders, and 2) how to judge if an amodal completion has succeeded. Our amodal completion method exhibits improved photorealistic completion results compared to existing approaches in numerous successful completion cases. And the best part? It doesn't require any special training or fine-tuning of models.

1. Introduction

Have you ever wondered how objects regularly occlude one another, yet we can effortlessly recognize and imagine their unoccluded appearance? Our visual system performs this task of amodal completion using the continuity and sym-metry of an object's shape [49] and everyday familiarity of the world [57]. Amodal completion, filling in hidden object parts, is a challenging AI task despite rapid advances in com-puter vision. This technology has many applications, such as in robotics, autonomous vehicles, and augmented reality. Figure 1.

Our method can recover the hidden pixels of objects in diverse images. Occluders may be co-occurring (a person on a surfboard), accidental (a cat in front of a microwave), the image boundary (giraffe), or a combination of these scenarios.