SRInpaintor: When Super-Resolution Meets Transformer for Image Inpainting

Recent image inpainting methods have achieved remarkable improvements by using generative adversarial networks (GAN). Most of them have been designed to produce plausible results from high-level semantic features using only high-resolution (HR) supervision. However, because abundant details are lost in large holes, it is difficult to simultaneously synthesize details while preserving structural coherence in HR space. Besides, the correlations between the inside and outside of the missing region play a critical role in transferring relevant known information to generate semantic-coherent textures, especially in patch matching-based methods. In this work, we present SRInpaintor which inherits the merits of super-resolution (SR) and transformer for high-fidelity image inpainting. The SRInpaintor starts from global structure reasoning with low-resolution (LR) input and progressively refines the local textures in HR space, constituting a multi-stage framework with SR supervision. The bottom stage recovers coarse SR results that provide structural information as an appearance prior, and is combined with the higher-resolution corrupted image at the next stage to render available textures for the missing region. Such a design can analyse the image from LR to HR with the increase of stages, enabling coarse-to-fine information propagation and detail refinement. In addition, we propose a hierarchical transformer (HieFormer) to model the long-term correlations between distant contexts and holes. By embedding it into a compact latent space in a cross-scale manner, we can ensure reliable relevant texture transformation and robust appearance consistency. Experimental results demonstrate the superiority of our method compared with recent state-of-the-art methods. Code will be available on https://github.com/lifengshiwo/SRInpaintor.

I. Introduction

Image inpainting focuses on completing the missing regions in a corrupted image with photo-realistic details and structures, which has been widely applied in many computer vision tasks, such as unwanted object removal, re-targeting, and image-based rendering [1]–[7].