Learning Adaptive Patch Generators for Mask-Robust Image Inpainting

In this paper, we propose a Mask-Robust Inpainting Network (MRIN) approach to recover the masked areas of an image. Most existing methods learn a single model for image inpainting, under a basic assumption that all masks are from the same type. However, we discover that the masks are usually complex and exhibit various shapes and sizes at different locations of an image, where a single model cannot fully capture the large domain gap across different masks. To address this, we learn to decompose a complex mask area into several basic types and recover the damaged image in a patch-wise manner with a type-specific generator. More specifically, our MRIN consists of a mask-robust agent and an adaptive patch generative network. The mask-robust agent contains a mask selector and a patch locator, which generates mask attention maps to select a patch at each step. Based on the predicted mask attention maps, the adaptive patch generative network inpaints the selected patch with the generators bank, so that it sequentially inpaints each patch with different patch generators according to its mask type. Extensive experiments demonstrate that our approach outperforms most state-of-the-art approaches on the Place2, CelebA, and Paris Street View datasets.

I. Introduction

Image inpainting [1], [2], also known as image completion, aims to recover the masked areas for a damaged or missing image with plausible pixel values based on the information of the known areas. This task plays a significant role in the field of image processing and has been widely used in various applications [3]–[5] such as photo editing, object removal, and old photo restoration [6]–[8]. Thus, it has drawn continuous attention for decades [9]–[13].