Storage-Computational Complexity Efficient Light Field Reconstruction

The point spread function (PSF) of plenoptic camera is verified to be spatial varying theoretically. Therefore, memory and time are consumed severely during the reconstruction of large-scale light field at the object plane where inversing the PSF matrix is needed. This problem will directly limit the spatial resolution of the object that can be handled. In this paper, a layered LU decomposition, partitioned Gaussian elimination and memory reusing method are proposed to reconstruct the light field for plenoptic camera. Layered LU decomposition together with partitioned Gaussian elimination makes a better use of computer's memory hierarchies and increases computing efficiency. The intra layer memory reusing method further reduces the memory consumption by in-place updating. Compared with existing methods, the proposed algorithm can reduce the memory consumption by the maximum of 1.85 times. It also provides the best trade-off between the computational complexity and memory consumption.

I. Introduction

Different from conventional cameras, plenoptic cameras can capture 4D spatial and angular information of light field via a single shot by inserting a micro lens array between the main lens and image sensor [1–4]. The 4D data can be used in depth estimation, view synthesis and saliency detection [5–7], etc. Image formation analysis is investigated in [8–12] to reconstruct the light field of the object, which can further benefit these applications. The formation of image and its corresponding PSF is given by $$\begin{equation*}i(x, y)=\displaystyle \int_{-\infty}^{+\infty}\int i_{\mathrm{o}}(x_{o},\ y_{0})h(x, y,x_{o},\ y_{0})dx_{o}dy_{0}, \tag{1}\end{equation*}$$ where and are the spatial coordinates at the object plane and sensor plane, respectively; and are the original light field information of the object and image data, respectively; is the impulse response of the corresponding plenoptic system, i.e. the PSF. Transforming this integration into discrete matrix form, the model in (1) is given as $$\begin{equation*}[I]=[\mathrm{H}][\mathrm{I}_{\mathrm{o}}], \tag{2}\end{equation*}$$ where I and are the column vectors which contain all the and , respectively; H is the PSF matrix that contains all the . The detailed arrangement is formulated as $$\begin{equation*}\left[ \begin{array} { c } { i ^ { 1,1 } } \\ { i ^ { 1,2 } } \\ { \vdots } \\ { i ^ { M , N - 1 } } \\ { i ^ { M , N } } \end{array} \right] = \left[ \begin{array} { c c c c c } { h _ { 1,1 } ^ { 1,1 } } & { h _ { 1,2 } ^ { 1,1 } } & { h _ { 1,3 } ^ { 1,1 } } & { \dots } & { h _ { M , N } ^ { 1,1 } } \\ { h _ { 1,1 } ^ { 1,2 } } & { h _ { 1,2 } ^ { 1,2 } } & { h _ { 1,3 } ^ { 1,2 } } & { \dots } & { h _ { M , N } ^ { 1,2 } } \\ { h _ { 1,3 } ^ { 1,3 } } & { h _ { 1,2 } ^ { 1,3 } } & { h _ { 1,3 } ^ { 1,3 } } & { \dots } & { h _ { M , N } ^ { 1,3 } } \\ { \vdots } & { \vdots } & { \vdots } & { \ddots } & { \vdots } \\ { h _ { 1,1 } ^ { M , N } } & { h _ { 1,2 } ^ { M , N } } & { h _ { 1,3 } ^ { M , N } } & { \cdots } & { h _ { M , N } ^ { M , N } } \\ { h _ { 1,1 } ^ { M , N } } & { h _ { 1,2 } ^ { M , N } } & { h _ { 1,3 } ^ { M , N } } & { \cdots } & { h _ { M , N } ^ { M , N } } \end{array} \right] \left[ \begin{array} { c } { i _ { o , 1,1 } } \\ { i _ { o , 1,2 } } \\ { \vdots } \\ { i _ { o , M , N - 1 } } \\ { i _ { o , M , N } } \end{array} \right] \tag{3}\end{equation*}$$ where MN is the spatial resolution of the image data on the sensor. The PSF is verified to be spatial varying theoretically, which indicates that the PSF matrix, H, is dense and large in scale [17]. To be specific, for a light filed image in size of 32803280, the size of its PSF matrix is (32803280)(32803280), which occupies almost 926TB memory storage space. It is surely too huge to be handled and becomes an obstacle for reconstructing light field of the object.