Structure-Guided Image Completion With Image-Level and Object-Level Semantic Discriminators

Structure-guided image completion aims to inpaint a local region of an image according to an input guidance map from users. While such a task enables many practical applications for interactive editing, existing methods often struggle to hallucinate realistic object instances in complex natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole region as well as the lack of a mechanism to enforce realistic object generation. In this work, we propose a learning paradigm that consists of semantic discriminators and object-level discriminators for improving the generation of complex semantics and objects. Specifically, the semantic discriminators leverage pretrained visual features to improve the realism of the generated visual concepts. Moreover, the object-level discriminators take aligned instances as inputs to enforce the realism of individual objects. Our proposed scheme significantly improves the generation quality and achieves state-of-the-art results on various tasks, including segmentation-guided completion, edge-guided manipulation and panoptically-guided manipulation on Places2 datasets. Furthermore, our trained model is flexible and can support multiple editing use cases, such as object insertion, replacement, removal and standard inpainting. In particular, our trained model combined with a novel automatic image completion pipeline achieves state-of-the-art results on the standard inpainting task.

I. Introduction

Recently, there have been increasing attention to and demand on guided image completion [1], [2], [3], [4], [5], [6], [7], [8], [9] for photo editing and creative expression. The aim of the guided image completion is to complete the missing region of an image according to an optional guidance map such as semantic label map [5], [6], edge map [1], [2], [7], [10], or colored pixels [8], [9]. Such tasks are shown to enable versatile image editing operations such as completion of large missing regions [1], [2], [10], [11], object removal [6], [12], [13], insertion [5], [6], [14], replacement [5], [6], [15] and manipulating the layout of an image [1], [4], [6], [7].