Spatial-Frequency Mutual Learning for Face Super-Resolution

Face super-resolution (FSR) aims to reconstruct high-resolution (HR) face images from the low-resolution (LR) ones. With the advent of deep learning, the FSR technique has achieved significant breakthroughs. However, existing FSR methods either have a fixed receptive field or fail to maintain facial structure, limiting the FSRperformance. To circumvent this problem, Fourier transform is introduced, which can capture global facial structure information and achieve image-size receptive field. Relying on the Fourier transform, we devise a spatial-frequency mutual network (SFMNet) for FSR, which is the first FSR method to explore the correlations between spatial and frequency domains as far as we know. To be specific, our SFMNet is a two-branch network equipped with a spatial branch and a frequency branch. Benefiting from the property of Fourier transform, the frequency branch can achieve image-size receptive field and capture global dependency while the spatial branch can extract local dependency. Considering that these dependencies are complementary and both favorable for FSR, we further develop a frequency-spatial interaction block (FSIB) which mutually amalgamates the complementary spatial and frequency information to enhance the capability of the model. Quantitative and qualitative experimental results show that the proposed method out-performs state-of-the-art FSR methods in recovering face images. The implementation and model will be released at https://github.com/wcy-cs/SFMNet.

1. Introduction

Face super-resolution (FSR), also known as face halluci-nation, is a technology which can transform low-resolution (LR) face images into the corresponding high-resolution (HR) ones. Limited by low-cost cameras and imaging con-ditions, the obtained face images are always low-quality, resulting in a poor visual effect and deteriorating the down-stream tasks, such as face recognition, face attribute analysis, face editing, etc. Therefore, FSR has become an emerging scientific tool and has gained more of the spotlight in the computer vision and image processing communities [20]. Figure 1.

Decomposition and reconstruction of face image in the frequency domain. (a) Denote face images; (b) are their amplitude spectrum; (c) show their phase spectrum; (d) present the reconstructed images with amplitude information only; (e) are the re-constructed images with phase information only.