Boosting Light Field Image Super Resolution Learnt From Single-Image Prior

In recent years, many deep learning networks are proposed for light field super resolution (LFSR). LFSR problem is essentially ill-posed since unknown detail information need to be predicted. Hence LFSR networks require plentiful content information (e.g., shape, color, texture) learned from sufficiently diverse scenarios. However, due to the high collection cost, existing light field datasets are in small size and have few scenarios, which could not meet the requirement and limit the performance of LFSR networks. To solve this problem, we proposed a novel framework to significantly boost their performance. Our main idea is to introduce valuable and plentiful content information from single images into LFSR networks as prior. Specifically, first, a view synthesis method is applied to add unreal disparity into single images, increasing the dimensionality of single images, hence to solve the problem of inconsistent data modalities. Then, we design the Scenarios-Content Introduction Module (SCIM) to effectively extract content feature from synthesized data. Finally, due to the added unreal disparity in the first stage, the features have severely pseudo information. Hence the Feature Attention Module (FAM) is proposed to discriminately select valuable information and combine this information into the LFSR network. Extensive experiments on six datasets validate the effectiveness of the proposed method, leading to a maximum gain of 0.439 dB. Our method can even boost SOTA networks to achieve higher performance.

I. Introduction

Light field (LF) imaging is a technology that simultaneously records the spatial and angular information of light, which has leading to profound progress in virtual realitys [1], [2], augmented reality and other emerging immersive multimedia applications [3], [4], [5], [6]. However, limited by sensor size, there is a trade-off between spatial and angular resolution [7], [8], [9], i.e., one could not obtain high both spatial and angular resolution, which seriously hinder its application [5], [10], [11], [12]. Therefore, how to improve the resolution of LF images, i.e. light field super resolution (LFSR), has attracted extensive research interest [13], [14], [15].