Pro-UIGAN: Progressive Face Hallucination From Occluded Thumbnails

In this paper, we study the task of hallucinating an authentic high-resolution (HR) face from an occluded thumbnail. We propose a multi-stage Progressive Upsampling and Inpainting Generative Adversarial Network, dubbed Pro-UIGAN, which exploits facial geometry priors to replenish and upsample ( $8\times $ ) the occluded and tiny faces ( $16\times 16$ pixels). Pro-UIGAN iteratively (1) estimates facial geometry priors for low-resolution (LR) faces and (2) acquires non-occluded HR face images under the guidance of the estimated priors. Our multi-stage hallucination network upsamples and inpaints occluded LR faces via a coarse-to-fine fashion, significantly reducing undesirable artifacts and blurriness. Specifically, we design a novel cross-modal attention module for facial priors estimation, in which an input face and its landmark features are formulated as queries and keys, respectively. Such a design encourages joint feature learning across the input facial and landmark features, and deep feature correspondences will be discovered by attention. Thus, facial appearance features and facial geometry priors are learned in a mutually beneficial manner. Extensive experiments show that our Pro-UIGAN attains visually pleasing completed HR faces, thus facilitating downstream tasks, i.e., face alignment, face parsing, face recognition as well as expression classification.

I. Introduction

With the increasing demand for social security, non-intrusive identity verification has become indispensable in daily life. As the face is one of the most frequently utilized biometric cues, it is highly desirable to obtain face images of high quality for providing essential information for identity verification. However, in real scenarios, the captured faces might not only be in low resolutions due to the long distance but also undergo occlusions caused by body parts or accessories, such as eyeglasses, scarves, etc. Due to the low resolution and occlusions, it becomes difficult, if not impossible, to extract useful knowledge to support downstream applications, such as face verification and facial attribute classification. Therefore, it becomes necessary to design advanced methods to hallucinate LR images with various occlusions.