LFSRDiff: Light Field Image Super-Resolution via Diffusion Models

Diffusion models have become a rising star in image super-resolution (SR) tasks. However, it is not trivial to apply diffusion models for light field (LF) image SR, which requires maintaining the high-quality visual appearance of each sub-aperture image (SAI) and the angular consistency between the different SAIs. This paper proposes the first diffusion-based LF image SR model, namely LFSRDiff, by incorporating the LF disentanglement mechanism and residual modeling. Specifically, we introduce a disentangled U-Net (Distg U-Net) for diffusion models, enabling improved extraction and fusion of the spatial and angular information in LF images. Furthermore, we leverage residual modeling in diffusion to learn the residual between the upsampled low-resolution and the ground truth high-resolution, which significantly accelerates model training and yields superior results compared to direct learning. Extensive experiments conducted on the five datasets demonstrate the effectiveness of our approach, which can produce realistic SR results and achieve the highest perceptual metric in terms of LPIPS. Code is publicly available at https://github.com/chaowentao/LFSRDiff.

I. Introduction

Light field (LF) cameras can simultaneously record the spatial and angular information of the scene through a single snapshot, making them valuable for various applications such as depth estimation [1]–[4], view synthesis [5], 3D reconstruction [6], and virtual reality [7]. However, due to the inherent trade-off between the spatial and angular in LF cameras, the spatial resolution of LF images is limited which imposes a constraint on the accuracy of subsequent applications. Hence, LF image super-resolution (SR) is an important research hotspot and there are numerous methods [8]–[21] have been developed to enhance the resolution of LF images, leading to significant advancements.