Video Lightening with Dedicated CNN Architecture

Darkness brings us uncertainty, worry and low confidence. This is a problem not only applicable to us walking in a dark evening but also for drivers driving a car on the road with very dim or even without lighting condition. To address this problem, we propose a new CNN structure named as Video Lightening Network (VLN) that regards the low-light enhancement as a residual learning task, which is useful as reference to indirectly lightening the environment, or for vision-based application systems, such as driving assistant systems. The VLN consists of several Lightening Back-Projection (LBP) and Temporal Aggregation (TA) blocks. Each LBP block enhances the low-light frame by domain transfer learning that iteratively maps the frame between the low- and normal-light domains. A TA block handles the motion among neighboring frames by investigating the spatial and temporal relationships. Several TAs work in a multi-scale way, which compensates the motions at different levels. The proposed architecture has a consistent enhancement for different levels of illuminations, which significantly increases the visual quality even in the extremely dark environment. Extensive experimental results show that the proposed approach outperforms other methods under both objective and subjective metrics.

I. Introduction

In order to resolve the problem of walking or driving in the dark environment, we may make use of visual aid (including small devices such as mobile phones), which is more easily or commonly available as compared with dedicated approaches such as infrared detection. Since each video composes of a sequence of images frames. Let us start with the problem of the image with bad lightening conditions. Images captured with insufficient illumination conditions usually have bad visual quality, such as low contrast, dim color, etc. Information among the low-light images faces substantial degradation that reduces its utility value. There could be possible solutions for taking photos in the low-light conditions, such as, using flash, increasing the sensitivity of the camera sensor (ISO) and taking photos with longer exposure time. However, these solutions have significant limitations: Flash may not be allowed in some public place, like cinema, museum, exhibition, etc. Higher camera sensitivity often brings noticeable noise in the dark regions. Longer exposure time is impractical for video capturing. Burst processing takes multiple low-light images under different exposures at a short time, and then combines them to obtain a large dynamic range. However, it cannot be generalized for enhancing low-light videos. Hence, it may be inevitable to obtain low-visibility images in low-light conditions. Enhancing such low-light images not only gives us a better visual quality but also gives benefit to vision-based systems, like autonomous driving, vision-based place recognition, etc. Figure 1.

Effect of our proposed method (the green and yellow circles show the significant improvement)