Multi-Scale Spatial-Angular Collaborative Guidance Network for Heterogeneous Light Field Spatial Super-Resolution

Light Field (LF) imaging captures the spatial and angular information of light rays in the real world and enables various applications, including digital refocusing and single-shot depth estimation. Unfortunately, due to the limited sensor size of LF cameras, the captured LF images suffer from low spatial resolution while providing a dense angular sampling. Existing single-input LF spatial super-resolution (SR) methods usually utilize the inherent sub-pixel information to recover high-frequency textures, but they struggle in large-scale SR tasks (e.g., $8\times $ ). Conversely, the heterogeneous imaging approach combining an LF camera and a 2D digital camera can capture richer information for effective large-scale reconstruction. To this end, this paper proposes a multi-scale spatial-angular collaborative guidance network (LF-MSACGNet) for heterogeneous LF spatial SR. Specifically, a context-guided deformable alignment module is first designed, which utilizes high-level feature information to achieve precise alignment between the low-resolution LF image and the 2D high-resolution image. Subsequently, a Transformer-driven spatial-angular collaborative guidance module is constructed to explore the spatial-angular correlation and complementarity. This allows for an effective fusion of the multi-resolution spatial-angular features. Finally, the SR LF image is reconstructed through a spatial-angular aggregation module. In addition, a multi-scale training strategy is adopted to subdivide the challenging large-scale SR task into multiple simple tasks to boost the SR performance. Experimental results on seven public datasets show that the proposed method outperforms the state-of-the-art SR methods in both quantitative and qualitative comparison, and exhibits favorable robustness to wide baseline LF images.

I. Introduction

Compared with traditional 2D imaging, light field (LF) imaging has the ability to capture the intensity and direction information of light rays, enabling scene reconstruction from different perspectives [1], [2], [3], [4]. Especially, the emergence of commercial LF cameras has brought LF into public view, thanks to their portability and ease of use. Currently, LF is widely used in multiple fields, including digital refocusing [5], depth estimation [6], three-dimensional reconstruction [7], semantic segmentation [8], and television display [9]. However, due to the inherent trade-off between spatial and angular dimensions [10], captured LF images suffer from low spatial resolution while maintaining a dense angular sampling. This defect seriously damages the visual quality of LF images and limits their performance in high-precision applications. Therefore, it is urgent to explore the LF spatial super-resolution (SR) approach.