Learn From Unpaired Data for Image Restoration: A Variational Bayes Approach

Collecting paired training data is difficult in practice, but the unpaired samples broadly exist. Current approaches aim at generating synthesized training data from unpaired samples by exploring the relationship between the corrupted and clean data. This work proposes LUD-VAE, a deep generative method to learn the joint probability density function from data sampled from marginal distributions. Our approach is based on a carefully designed probabilistic graphical model in which the clean and corrupted data domains are conditionally independent. Using variational inference, we maximize the evidence lower bound (ELBO) to estimate the joint probability density function. Furthermore, we show that the ELBO is computable without paired samples under the inference invariant assumption. This property provides the mathematical rationale of our approach in the unpaired setting. Finally, we apply our method to real-world image denoising, super-resolution, and low-light image enhancement tasks and train the models using the synthetic data generated by the LUD-VAE. Experimental results validate the advantages of our method over other approaches.

1 Introduction

Image restoration aims to recover the underlying clean image ${\mathbf {x}}$ from the corrupted observation ${\mathbf {y}}$, $$\begin{equation*} {\mathbf {y}}= {\mathcal {T}}({\mathbf {x}}) + {\mathbf {n}}, \tag{1} \end{equation*}$$ y=T(x)+n,(1) where ${\mathbf {n}}$ represents the noise, and ${\mathcal {T}}$ represents the degradation operation. This task is one of the fundamental problems in computer vision and has been extensively studied for decades [3], [4], [5]. In recent years, deep learning has achieved astonishing success in image restoration problems, such as image denoising [6], [7], [8] and super-resolution [9], [10], [11], [12]. However, the success of these methods requires large quantities of paired training data, and the restoration performance is sensitive to the degradation types [13], [14], [15]. For example, one Gaussian denoising network usually performs poorly for real-world noisy images due to the noise discrepancy between Gaussian noise and real-world noise [16]. Meanwhile, collecting paired training data for real-world image restoration is cumbersome and expensive due to the complex camera image signal processing (ISP) pipeline [17], [18], [19]. The aforementioned problems make real-world image restoration a challenging task. On the other hand, unpaired data broadly exists and is easily accessible in many situations. For example, it is easy to obtain many images of different resolutions or noisy and clean images through the internet [20]. Consequently, designing deep learning methods with unpaired data is of significant research importance and deserves deep exploration.