Light-Field Camera for Fast Switching of Time-Sequential Two-Dimensional and Three-Dimensional Image Capturing at Video Rate

In this article, we propose a time-sequential switching light-field (LF) camera for an alternative image capture of the high-resolution two-dimensional (2-D) images and three-dimensional (3-D) LF elemental images as additional functionalities. For image data acquisitions of both the 2-D and 3-D LF imaging of moving objects at a video frame rate (or even higher frame rate up to approximately 1000 f/s), a polarization-dependent-switching microlens array (MLA) is implemented in the LF camera system instead of a conventional passive-type MLA. By controlling the incident polarization conditions using an electrically fast-switching liquid crystal layer, the imaging mode can be time-sequentially switched quite rapidly (switching times of approximately 220 and 290 μs for the mode conversions from the 3-D LF to the 2-D mode, and the reversal mode change, respectively). Using the elemental image sets sampled from the alternating time-sequential imaging results, either the directional-view images or depth-refocused images can be reconstructed and provided at a moving picture frame rate. The depth-refocused images are possible for a wide depth range from 25 to 350 cm. Directional views of 22 × 22 and 9 × 9 in portions can be reconstructed for the single-shot image capture and the time-sequential video-rate image capture, respectively.

SECTION I. Introduction

Light-field (LF) imaging techniques acquire positional ray information from object points on directional vectors, as well as intensities utilizing periodic ray sampling, usually by the introduction of a microlens array (MLA) [1]–​[5]. The use of the elemental image sets captured through the MLA enables the extraction of three-dimensional (3-D) object information. This facilitates the reconstruction of the directional-view or depth-refocused images based on full parallax information [6]–​[12]. However, instead of simply acquiring additional ray direction information through the use of the MLA, contrary to the conventional two-dimensional (2-D) imaging systems, the spatial resolution of the reconstructed image results inevitably degrade at the fixed pixel resolution of an image sensor array. The spatial resolution of the reconstructed LF images can be increased by increasing the MLA density, but then the angular resolution degrades with the decrease in the number of image sensor pixels matching each elemental lens.

Currently, users are familiar with high-density images formed in full high definition, such as those produced when capturing images with digital cameras or camera modules in cellular phones. As such, the low resolution of LF imaging results has proved to be an insurmountable obstacle in attempting to apply the LF imaging scheme to commercialized personal mobile products, even though the scheme can provide very discriminating 3-D experiences to users. To enhance the resolution issues associated with the reconstructed images from the LF cameras, a modified graphic processing unit (GPU) was proposed by analyzing the ray space model [13]. Other proposed solutions include a mechanically shifting MLA scheme for increasing periodic spatial sampling [14], GPU and field programmable gates arrays [15]. Nevertheless, the resolution of the reconstructed images taken with an LF camera has remained quite low compared to the resolution of conventional 2-D images obtained without an MLA.

To overcome this barrier to commercialization, it is necessary to find a way for the LF imaging functions such as directional views or depth-refocused images to be provided as additional functionalities without sacrificing the conventional high-resolution 2-D image capture functionality expected of imaging devices. Generally, the LF cameras consist of a main lens, a passive-type MLA, and an image sensor array. In this configuration, the use of a switchable MLA can offer a solution for providing both functional 3-D LF imaging and high-resolution 2-D imaging. For application to portable devices, the switchable MLA should be appropriate for implementation in a compact module. In addition, considering intensifying current trends for the moving pictures, the LF imaging module utilizing the switchable MLA should be able to provide both the required functional 3-D LF imaging and high-resolution 2-D imaging simultaneously, at a frame rate appropriate for the image capture of moving objects.

The use of a liquid crystal (LC) based active-switching lens is a promising approach which can be implemented in a very thin and compact optical module [16]–​[20]. Considering device structures and their operational principles, there are several types of LC-based switchable lenses. One type utilizes the field-switching gradient refractive index (GRIN) lens profiles of the LC layer by introducing patterned electrodes [21]–​[23] or surface curvatures [24]. Another type utilizes field-induced switching between the focused state and the defocused state, or field-induced variable focusing properties of the LC-based switchable lens. Several varieties of 3-D applications have been achieved in this manner, including autostereoscopic 3-D displays [25]–​[27], 3-D LF displays [28]–​[30], LF imaging systems [7], 3-D LF endoscopes [31], [32], and 3-D LF microscopes [10], [11]. Recently, to develop a 2-D to 3-D switchable LF camera, an LC-based active-switching MLA was implemented for an LF imaging system [31]. In this article, the GRIN profile in an LC layer was obtained using periodic distributions of patterned electric fields generated between a pair of concave and planar electrodes. According to field-induced LC reorientations, the defocusing state and the periodic focusing state could be switched to select between the 2-D and 3-D LF imaging modes. Nevertheless, due to the sluggish nature of the active-switching MLA, simultaneous image capture of the 2-D and 3-D information for moving objects proved impossible to attain.

In this article, we present an LF camera system which can provide both fast switching time-sequential image capture of the high-resolution 2-D images and functional 3-D LF imaging. An extremely fast-switching dynamic exists between the defocused state and the periodic ray-sampling of the focused state of the liquid crystalline reactive mesogen (RM) based MLA operated by control of an incident polarization state. This rapid switching property enables the 2-D imaging and 3-D imaging used for the LF image reconstruction to be acquired time sequentially at a frame rate greater than the moving picture frame rate for capturing moving objects. In Section II, the schematics and the operational principles of the proposed LF camera system using the RM-based polarization-dependent MLA (PMLA) will be presented together with the fabrication procedures of the PMLA. In Section III, experimental results of the elemental image capture and the LF image reconstruction are demonstrated and discussed. In Section III-A, the LF image reconstruction capability of the directional views and the depth-refocused images are demonstrated and discussed with reference to some supplementary videos. In Section III-B, the experimental results of the time-sequential image capture of the high-resolution 2-D imaging and the functional 3-D LF imaging at a moving picture frame rate for moving objects are presented. With the presented device scheme, either the high-resolution 2-D moving picture or the user-selective view (directional view or depth-refocused view) of the LF moving picture can be provided; this will also be discussed with reference to some supplementary videos. The conclusion and perspective of the presented time-sequential fast switching LF camera are provided in Section IV.

SECTION II. Schematics and Operational Principle of Time-Sequential 2-D and 3-D Image Sampling LF Camera Utilizing Switchable MLA

A. Schematic Diagram of Time-Sequential LF Image Sampling Using PMLA

Fig. 1 shows a schematic diagram of the LF camera system using the polarization-dependent-switching MLA. We set the distance between the main object lens (AF Nikko f/1.4D, Nikon Ltd.) and the periodic angular ray-sampling PMLA to be the focal length amount $f_{l}$ of the main object lens. For switching the PMLA between the defocused state and the periodically focused state by control of incident polarization, a polarizer and a fast-switching polarization control LC layer were installed between the main lens and the PMLA. In the scheme, as shown in Fig. 1, the transmission axis of the polarizer is parallel with the y-axis. The initial LC alignment azimuth direction of the polarization-switching LC cell was set to be 45° with respect to the x and y axes. The image sensor (EOS 80D, Canon Inc.) was placed at a distance from the focal length g behind the PMLA. In our experiment, the elemental image sets from the PMLA were captured through a relay lens (EF 100 mm f/2.8 L Macro IS USM, Canon, Inc). To minimize image overlapping on the image sensor plane between the neighboring elemental images, the f-number (f/16) of the PMLA in the periodic focused state was set to match that of the main lens. The elemental images or high-resolution 2-D images were captured using the image sensor with the relay lens under the f/2.8 f-number and 1/50 s shutter speed conditions. Table I details the parameters of the optics implemented in the proposed LF camera system.

Fig. 1.

Schematic diagram of the time-sequential 2-D and LF 3-D image sampling camera operated by the PMLA.

TABLE I Parameters of Optics Implemented for an LF Camera System

B. Fabrication Process and Operational Principle of PMLA

For the time-sequential 2-D and 3-D LF image acquisitions for a moving picture at the video frame rate, a high-speed switching MLA supporting more than 50 frames/s (f/s) is required. For this purpose, the LC-based switchable lens utilizing field-induced control of the GRIN lens LC profiles is inappropriate; the field-on or field-off LC-switching dynamics inevitably become slow due to the complex LC geometries and thick LC layer conditions, especially for the short focal length and the small f-number lens design required in the imaging systems [16], [20]. In this article, the PMLA was implemented for fast switching of the focusing–defocusing properties that are controlled by an incident polarization state rather than by the LC molecule reorientation within the switchable lens layer [30]. In this case, with a fast-switching polarization control layer, the time-sequential 2-D and 3-D LF image capture can be achieved at a rate above the video capture frame rate.

Fig. 2 shows a schematic diagram of the fabrication procedures of the implemented PMLA. To sufficiently reduce the thickness of the PMLA to one which promises to be viable for a compact module, the RM-based MLA was prepared as the PMLA. One of the other merits of the RM-based lens is that the focusing and defocusing optical properties determined by the incident polarization can be precisely designed to produce ideal phase profiles by using the refractive index and the surface curvature conditions [33]. These polarization-dependent controls of the focusing and defocusing phase profiles can also be preserved well after converting the RM layer to a solidified film with UV-induced polarization. Owing to this feature, the PMLA structure presented in this article can be reliably developed with entirely cost-effective fabrication procedures, as shown in Fig. 2.

Fig. 2.

Fabrication process of the PMLA. (a) Imprinting process to form the periodic planar-concave surface topology using the MLA template: spin-coating of UV-curable resin and laminating the MLA template, UV-induced polymerization, and detaching the MLA template. (b) Surface treatment for the RM alignment: hydrophilic surface modification by the UVO pretreatment, spin-coating and baking of the PVA layer, and rubbing process for the bottom–up RM alignment effect. (c) Preparation of the top–down RM alignment film substrate: spin-coating and baking of the PVA layer and applied the rubbing process. (d) Preparation of the planar-convex birefringent RM layer on the planar-concave MLA surface: drop-casting of the RM, lamination of the top–down alignment film, UV-induced polymerization of the RM layer after the RM alignment stabilization, and detaching of the top film.

To implement the polarization-dependent-switching MLA, we used a square-arranged, bed-pillow-shaped, planar-convex MLA substrate as an imprinting template. The substrate had a lens pitch of 100 μm and a focal length of 800 μm. The planar-concave MLA is made with an optically isotropic material by using a UV-nanoimprinting process with a planar-convex MLA template, as shown in Fig. 2(a). For the imprinting material, a UV-curable resin of NOA81 ($n_{p}= 1.51$, Norland Products Inc.) was used. A UV-ozone (UVO) treatment was applied for 2 h to the planar-concave MLA surface made with NOA81 to provide a hydrophilic surface condition, as shown in Fig. 2(b). On the UVO-treated surface, a polyvinyl alcohol (PVA) solution was spin-coated and baked at 90 °C for 30 min as the RM alignment surface. To achieve a uniform alignment condition for the RM layer, the PVA layer was rubbed with a rubbing machine. To achieve a well-aligned RM layer on the planar-concave MLA surface, a film substrate for the top–down RM alignment enhancement was additionally prepared, as shown in Fig. 2(c), with a similar surface treatment procedure to that shown in Fig. 2(b). On the PVA-coated planar-concave MLA surface, RM (RMM727, the ordinary and extraordinary refractive indices are: $n_{o}= 1.51$, $n_{e}= 1.68$, Merck) was drop-cast and the top alignment film was laminated onto it. After thermal treatment for the alignment of the RM film between two substrates, the RM layer was polymerized by UV exposure. Finally, the PMLA was obtained, as shown in Fig. 2(d), by detaching the top–down alignment film.

Fig. 3(a) and (b) shows the operational principle of the polarization-dependent-switching MLA, where the PMLA is stacked with the polarization-switching layer. For fast-switching dynamics, the optically compensated bend (OCB) LC mode was applied to the polarization-switching LC layer [34]. In Figs. 2 and 3, the thickness of the PMLA was about 0.73 mm including the underlying glass substrate. The thicknesses of the polarization-switching layer and the front polarizer film were about 1.41 and 0.15 mm, respectively. The total thickness of the stacked structure was below 2.5 mm including the adhesive layers between the components. The presented scheme was achievable in a highly compact and lightweight optical module. When the incident polarization state determined by the applied field condition of the underlying polarization-switching LC layer is parallel to the RM alignment direction of the PMLA, periodically focused beam patterns are obtained as shown in Fig. 3(a) and (c). This is because $n_{e}$ of the planar-convex RM layer is higher than $n_{p}$ of the planar-concave isotropic polymer layer. For this state, the polarization-switching layer is operated as the half retardation wave plate and the PMLA acts as the 3-D LF mode capturing the elemental image sets, which were obtained at the applied voltage of 2.4 V_p in our experiment. When the polarization state is switched to be parallel to the no axis of the RM layer, the incident rays are not refracted with propagation through the PMLA, as shown in Fig. 3(b) and (d). This is because of the index matching condition between the n_o of the RM layer and the n_p of the isotropic polymer layer, despite an interfacial curvature between the two layers. For this state capturing the high-resolution 2-D images, a high voltage of 30 V_p was applied to the polarization-switching layer, which was operated as a polarization-preserving optical c-plate. In our switching operation between the 2-D and 3-D LF mode image capturing, the incident polarization state before the PMLA needs to be precisely controlled between two orthogonal linear polarizations by the polarization control layer. When we evaluated the degree of polarization controlled by the voltage-dependent polarization-switching layer [35], the Stokes parameter values of S₁ after the polarization-switching layer were measured as −0.9873 and 0.9871 at 30 V_p (2-D mode) and 2.4 V_p (3-D LF mode) operations, respectively. This means that the imaging crosstalk between the 2-D and 3-D LF modes, which can be caused by nonideal polarization control, could be effectively minimized in our experiment, as shown in the captured original images of Fig. 3(h) and (i). When we evaluated the spectral transmittance of the stacked structure of the polarization-switching layer and the PMLA layer, the transmittance level at a shorter wavelength was slightly lower than that at a longer wavelength because of the light absorption effect at the UV-cured RM layer of the PMLA and the transparent electrode layers of the polarization-switching layer. However, over the whole visible range (470–700 nm), the stacked optical module was still highly transparent with the average transmittance level of 90.04%.

Fig. 3.

(a) and (b) Schematic diagram showing switching behavior between the 2-D periodic focusing and defocusing states according to the incident polarization state electrically controlled by the polarization-switching layer. (c) and (d) Focused and defocused beam patterns, respectively, observed by the optical microscope while changing the incident polarization state. (e) RM texture observed by the polarizing optical microscope, where “P” and “A” denote the transmission axes of the crossed polarizers and “R” denotes the alignment direction of the RM layer of the PMLA. (f) Relative phase profile obtained along the yellow dashed line of (e). (g) MTF values measured for the MLA template and the PMLA using the 1951 USAF resolution chart. (h) and (i) Captured elemental image arrays for the LF 3-D mode and the high-resolution 2-D mode, respectively, and their enlarged positional pictures.

The distribution of the positional relative retardation of the PMLA can be seen by observing the light transmittance by placing the PMLA between the crossed polarizers, as shown in Fig. 3(e), where the optic axis of the liquid crystalline polymer (LCP) RM layer is oriented at 45° with respect to the transmission axes of the crossed polarizers. Although the fabricated RM-based PMLA has almost 100% fill-factor MLA condition, each elemental lens exhibits ideally symmetric concentric fringe patterns, showing that there is no RM layer alignment defect. This was achieved due to the top–down and bottom–up alignment effects during the thermal annealing process performed for the RM alignment stabilization [33]. To obtain the PMLA with the f-number condition of f/16, a very thin RM layer (maximum thickness of 4 μm on the interfacial curvature of the planar-concave isotropic polymer layer) was required. Obviously, the thin RM thickness condition in our experiment also positively contributes in obtaining highly uniform RM alignment. For a PMLA with a longer pitch condition or a smaller f-number condition of the elemental lens, the maximum thickness of the RM layer on the curvature needs to be increased. When an RM thickness of more than about 100 μm is required for a lens design with a single interfacial curvature, a Fresnel-type surface curvature is more appropriate to reduce the RM thickness for a uniform RM alignment [36]. Fig. 3(f) shows the measured relative phase retardation profile, which was obtained along the yellow line in Fig. 3(e). The average value of discrepancy in the relative phase profiles between the measured and ideal phase curves was evaluated as 0.08π rad, as shown in Fig. 3(f). The measured relative phase profile for the PMLA was in good agreement with the ideal one [33].

The imaging performance of the PMLA was evaluated by obtaining the modulation transfer function (MTF) curve using the 1951 USAF resolution chart. At this measurement, an objective lens with a numerical aperture of 0.15 was additionally used. For the MTF evaluation of the PMLA, the polarization state for the periodic focusing was used as the incident beam condition. For comparison, the MTF curve for the passive-type planar-convex MLA template used for the isotropic planar-concave MLA surface during the nanoimprinting procedure in our experiment was also evaluated, as shown in Fig. 3(g). Under the 8 lines/mm imaging condition, the MTF values for the PMLA and the passive MLA template were evaluated as 0.73 and 0.64, respectively. For this characterization, the MTF values for the evenly spaced nine sampling areas (1.5 × 1.5 mm for each sampling region) within the total area (4 × 4 cm) of each lens array were measured and their average values were obtained. The MTF value of the PMLA was slightly higher than that of the passive MLA because the f-number of the PMLA (f/16) becomes larger than that of the passive MLA template (f/8) at the same surface curvature profiles developed by the UV-nanoimprinting process. Without the PMLA and with the objective lens only, the MTF value was measured as 0.81, which indicated that the fabricated PMLA works well in the 3-D mode. The standard deviation amount of the MTF values measured at the nine sampling areas of the PMLA was sufficiently low at 0.012, which exhibited excellent uniformity as the periodic imaging component for capturing the elemental image sets. When we changed the incident polarization condition as the orthogonal state, the MTF value measured with placing the PMLA operating as the 2-D imaging mode was 0.77, which means that the ray distortions at the PMLA can be effectively minimized at the 2-D mode operation for the high-resolution 2-D imaging.

Fig. 3(h) and (i) shows the captured images recorded by the image sensor when the PMLA was operated in the 3-D LF mode and the 2-D mode, respectively, by the incident polarization control of the polarization-switching layer, as shown in Fig. 1. The enlarged images for the different depth positions (regions i, ii, iii, and iv) of the objects are also presented in Fig. 3(h) and (i). All the enlarged pictures of Fig. 3(h) were clearly obtained, although they were captured for the objects located at the different depth positions. This shows that the periodic directional angular ray-sampling was well achieved by setting the PMLA to the same f-number condition as the object main lens in the LF imaging system. After the LF image reconstruction, the user-selective directional-view images or depth-refocused images can be obtained by using the obtained elemental image sets, as shown in Fig. 3(h). Contrarily, in the case of the 2-D mode imaging shown in Fig. 3(i), only the enlarged picture for region iv, where the main lens was focused, was clear. The images for the other regions were blurred because of the out-of-focus effect outside the depth of field (DOF) of the main lens.

SECTION III. Experimental Results and Discussions

A. Image Reconstruction of the LF Camera for Directional and Depth-Refocused Views

Fig. 4(a) shows the arrangement of the objects used for characterization of the directional-view image reconstruction from the single-shot image capture using the implemented LF camera. Due to the operational principles of the LF camera, the view disparity increases for objects placed far from the focal plane of the main lens [11], [14]. In our measurement, the objects for i and iv were placed on an axis parallel to the camera, and the objects for ii and iii were positioned to the right and left of the camera view direction, respectively. The index definition for each reconstructed directional-view image is indicated in Fig. 4(b) according to each view's position. The resolution of each reconstructed directional-view image is 94 × 103 pixels and the total number of the reconstructed directional views is 22 (horizontal) × 22 (vertical), as listed in Table II. Fig. 4(c) shows some examples of the reconstructed directional-view images, sampled to see the disparity effects along the vertical and horizontal directions, where the red- and blue-boxed regions within each image show the vertical and horizontal parallax clearly. The angular resolution of each reconstructed directional-view image captured by the implemented LF camera system was almost 1°. The reconstructed results for the different sets of the directional views are presented with the GIF images in the supplementary video, Video S1.

Fig. 4.

Directional-view images reconstructed from the elemental image arrays acquired by the implemented LF camera. (a) Arrangement of the real objects used for capturing the elemental images. (b) Index definition for the positional elemental image sets. (c) Examples of the reconstructed directional-view images.

TABLE II Key Parameters of an Implemented LF Camera System

The digital depth-refocusing function from the elemental image sets is one of the features distinguishing 3-D LF imaging from conventional 2-D imaging. Digital refocusing is a technique for regenerating images such that the result is focused on a depth plane different from that of the initial optically focused depth plane. This digital refocusing function is also obtainable from parallel multishot images from relatively bulky optical systems based on multiple cameras [37], [38]. Compared to these approaches, the 3-D LF imaging system can be constructed in an extremely compact and lightweight optical module even in our switchable LF imaging system with the switchable focusing units as explained in Section II-B. In addition, the 3-D LF imaging method based on the single image sensor and the single-lens unit does not lead the device or image calibration issues that originate from imaging discrepancies of imaging properties acquired at different imaging units in multiple-camera-based imaging systems and undesired packaging variation under gap conditions between each imaging module [37], [38]. In conventional digital cameras, the sharpness of a captured image has a tradeoff relationship with the aperture size or f-number of the main lens placed in front of the image sensor [7]. In this case, considering the moment of image capturing of a moving object, the exposure time of the digital camera should be short, and therefore, the aperture size should be larger to allow enough light to be gathered. However, the DOF results in the captured images become narrower because of the increased aperture size. The narrower DOF conditions mean that most of the depth-range images captured out of the optimum DOF plane become blurred. Conversely, for clear image capture of objects over a wide depth range, the aperture size of the digital camera should be smaller. In this case, the exposure time should be longer to allow enough light to be gathered. This longer exposure time means that the digital camera cannot capture clear images of moving objects. This tradeoff limits the acquisition of a sharp image of moving objects in the conventional 2-D mobile cameras constructed with a simple combination of lenses. On the contrary, with conventional imaging systems where the image sensor is located at the image plane of the objective lens, a floating volume image, formed at an intermediate image plane between the objective lens and the MLA, is recorded through the MLA at the image sensor as the elemental image array with parallax information [10], [11]. In this LF imaging system, the reconstructable depth range can be extended by the DOF of the MLA, which can be increased according to the condition of separation distance between the intermediate image plane and the MLA [39], [40]. Thus, the depth-refocusing function of the LF camera, utilizing the ray information from the elemental image sets, can be an effective solution to capture and reconstruct 3-D volume objects distributed in a large depth range.

Fig. 5 shows the experimental results after applying the digital depth-refocusing process using the elemental image sets captured with the proposed LF camera. To obtain the depth-sliced images reconstructed at a different depth from the single-shot image capturing of the elemental image array, the computational integral imaging reconstruction algorithm, reported by Kim et al., was used [41]. The object arrangements and the positional distance conditions are shown in Fig. 5(a). The distances of the four objects (i, ii, iii, and iv) from the LF camera were 25, 75, 100, and 350 cm, respectively. Compared with the highly blurred 2-D image [see Fig. 5(b)], except for the depth plane object of iv, the digitally depth-refocused images [see Fig. 5(d)] reconstructed with the LF elemental image sets of Fig. 5(c) can provide sharp image results for both the furthest and nearest objects. In the implemented LF imaging system, virtual synthetic image planes are achievable for the depth-refocusing over depths ranging from 25 to 350 cm, as shown in Fig. 5(d) and supplementary video, Video S2. Despite the unique functionality of the switchable image acquisitions between the high-resolution 2-D and 3-D LF modes, owing to the excellent optical property of the elemental image acquisition and its ideal switching capability of the demonstrated PMLA, the experimental reconstruction results of the presented LF camera as well as the characteristics listed Table II showed good features comparable with those of the previously reported LF cameras constructed with the passive-type MLA enabling 3-D LF imaging only [42], [43].

Fig. 5.

Depth-refocused images reconstructed from the elemental image arrays acquired by the implemented LF camera. (a) Arrangement of the real objects used for capturing the elemental images. (b) High-resolution 2-D image obtained at the defocused state of the PMLA. (c) 3-D LF image obtained at the periodically focused state of the PMLA. (d) Examples of the reconstructed depth-refocused images.

B. Time-Sequential 2-D and 3-D Information Acquisition at Video Rate With the LF Camera

In this section, the experimental results on the time-sequential acquisition of the high-resolution 2-D and functional 3-D LF images at a moving picture frame rate, and the moving pictures reconstructed for the directional-view or depth-refocused movies will be presented. In our scheme, the switching speed for the time-sequential alternative image capture is determined by the polarization-switching dynamics of the polarization control LC layer. Fig. 6 shows the schematic diagram used to characterize the response times required for switching between two orthogonal linearly polarized states in the polarization control layer. For the switching dynamics characterization of the polarization-switching layer, the signal waveform for driving the polarization-switching layer was generated using the data acquisition (DAQ, SCB-68A, National Instruments Co.) board controlled by LabVIEW programing. The analog output signal of the DAQ board was amplified using the voltage amplifier (A400, FLC Electronics) to obtain sufficient voltage amounts for the operation of the polarization-switching layer.

Fig. 6.

Measurement optical set-up for analyzing the switching dynamics of the polarization-switching layer.

The upper graph of Fig. 7(a) shows the signal waveform applied to the polarization control layer, where the higher and lower voltages at 1 kHz are applied alternatively every 20 ms for characterization of the switching dynamics. The lower graph in Fig. 7(a) shows the time-response characteristics of the polarization-switching layer when the driving voltage waveform is applied. The electro-optic properties of the polarization-switching layer were retained well for each frame time duration of the 2-D and 3-D modes at the same operating frequency of 1 kHz without causing the voltage holding problem [44]. In Fig. 7, the variation by time of the light transmittance level was measured using a photodetector connected to a digital oscilloscope (DSO1052B, KEYSIGHT, Ltd). When the applied voltage is 2.4 V_p, the LC layer acts as a half retardation waveplate, changing the polarization state incident on the PMLA for the 3-D LF mode image capturing. For the half retardation LC cell condition to change the incident polarization into an orthogonal state, the OCB LC cell requires a low voltage of 2.4 V_p to apply to prevent the slow LC-switching dynamics from the bend-to-splay LC relaxation [34]. In this case, the exit beams from the polarization control layer pass through the second polarizer. Conversely, when the applied voltage is 30 V_p, the LC layer is switched to an optical c-plate for 2-D image capture and the exit beams are blocked by the analyzer in our switching dynamics characterization, as shown in Fig. 6. To quantitatively analyze the switching times, Fig. 7(b) is presented for the moments in which the 2-D and 3-D LF mode switching occur. It can be seen that the measured switching times from the 3-D LF mode to the 2-D mode and for the reversal mode change were 220 and 290 μs, respectively. These measured response time values are fast enough to implement the proposed alternative switching LF camera system for capturing moving objects. In Fig. 7(c), the switching dynamics of the polarization control layer driven by the operating waveform are presented for the time-sequential image acquisition at a frame rate of 1000 f/s. For this switching dynamics characterization under a faster switching waveform, the higher voltage of 30 V_p (2-D mode) and lower voltage of 2.4 V_p (3-D LF mode) operated at 10 kHz were applied alternatively every 1 ms. The result, obtained at 20 times faster operation than Fig. 7(a), also shows a sharp light transmittance variation according to changes in waveform. The experimental results show that the proposed system can be promisingly applied to high-speed LF camera applications by adopting a high-speed image capture sensor module.

Fig. 7.

Switching dynamics of the polarization-switching layer. (a) Normalized light transmittance of the polarization-switching layer operated at 50 f/s waveform, which is measured under the crossed polarizers for characterizing the switching dynamics between two orthogonal polarization states. (b) Enlarged pictures of (a), showing the field-on and field-off response times. (c) Response time characteristics with the operating waveform for 1000 f/s.

Using the LF image camera implemented in this article, the time-sequential alternative image capture of the high-resolution 2-D images and the 3-D LF elemental images was performed at a video capture speed of 50 f/s, as shown in Fig. 8. For the moving object used for characterizing the image capture and the LF-image reconstruction properties, a miniature model of Eiffel tower suspended from the ceiling was used, rotated in the clockwise direction by approximately 15 °/s. A static object of the cube or the three static objects shown in Figs. 4 and 5 were located behind the floating object.

Fig. 8.

Time-sequential image acquisition for the high-resolution 2-D information and the LF 3-D elemental image information, which is measured at the video frame rate for the moving object (with rotating the miniature of Eiffel tower hanging from the ceiling). The operating waveform applied to the polarization-switching layer is coplotted.

Fig. 8 shows the operational waveform applied to the polarization control layer and the time sequentially captured frame images where the examples of the video-frame image acquisition at the moment sampled every 20 ms were presented. For this experiment, the operational waveform to control the polarization-switching layer was generated using the same driving units as those employed for the switching dynamics characterization shown in Fig. 7. Owing to the fast-switching properties of the polarization-switching layer, every frame time image could be acquired clearly at a video capture rate of 50 f/s without image overlapping between the different modes (the high-resolution 2-D imaging mode and the 3-D LF mode for the elemental image capture) of the subsequent moving frames, as shown in Fig. 8. The moving picture result showing the time-sequential alternative 2-D and 3-D LF imaging at the video frame rate is presented in the supplementary Video S3. Using the captured moving picture frames, the odd and even frames could be alternatively resampled for preparation of the video rate capture of the 2-D image and the LF elemental images, respectively. The odd-framed moving pictures can be used for providing the high-resolution 2-D video, and the even-framed ones can be used for image reconstructions of the directional-view or the depth-refocused videos. The odd- and even-framed videos are shown in the supplementary videos, Video S4 and Video S5, respectively. In this time-sequential video-rate image capture, the frame image resolution of the 2-D and 3-D LF images was the 990 × 774 pixels, and was limited by the maximum the image capture resolution of the image sensor utilized in our experiment, operated in the video frame rate, as listed in Table II.

After the LF image reconstruction using the results on the even-framed image capture processing, the directional-view or the depth-refocused videos can be provided. Fig. 9 shows the examples of the high-resolution 2-D images and the LF elemental images sampled at the moment of the moving picture frames at intervals of 1 s and the LF image reconstruction results for both the directional views and the depth-refocused views. In our time-sequential image capture at the video frame rate, the number of the reconstructed directional-view images was 9 × 9 views along the horizontal and vertical directions, as listed in Table II. In Fig. 9, the image resolution of the reconstructed directional views was the 110 × 87 pixels, also listed in Table II. The results of the digitally refocused video, obtained after the LF frame-image reconstruction, are presented in the supplementary Video S6.

Fig. 9.

Examples of the time-sequential image acquisitions for the high-resolution 2-D information and the LF 3-D elemental image information. At each instant, the elemental images acquired at the moving picture frame rate are reconstructed for the directional-view images and the depth-refocused images.

SECTION IV. Conclusion

In this article, we proposed a 3-D LF camera scheme which could also provide the high-resolution 2-D imaging results at a moving picture frame rate achieved by introducing the polarization-dependent-switching LCP-based MLA for periodic angular ray sampling in the LF imaging scheme instead of the conventional passive-type MLAs. As the properties presented for switching between the periodic focused states and defocused states were obtained by an incident polarization condition applied to the PMLA and electrically controlled by the fast-switching polarization control layer without any mechanical moving parts, the mode conversion in alternating between the 2-D imaging states and the elemental imaging states could be achieved rapidly and was applicable to the time-sequential alternative image capture at a video frame rate. The switching time measurement results showed that the presented imaging system could achieve high-speed video capture at a rate of nearly 1000 f/s. The proposed LF camera could reconstruct the directional-view and digitally depth-refocused images as well as retaining the image capture function of the high-resolution 2-D images obtainable from the conventional digital cameras. We expect that the high-resolution 2-D images or moving-frame images captured in the 2-D mode could also be utilized for enhancing the spatial or angular resolution of the LF elemental images after LF image computations [13], [15]. The presented LF imaging scheme, switchable over the video frame rate, could be implemented in such a compact module that it was widely applicable to several types of 3-D imaging applications, including bio-imaging microscopes, miniaturized endoscope systems, and mobile camera modules. Especially, the presented features of the time-sequential fast speed image capturing capability of the high-resolution 2-D mode and functional 3-D LF mode would be highly attractive for applications on robot vision systems and smart flying drones, which require real-time 3-D information on the environmental situations for automatic self-motion control [45], [46].