Multiframe-Based High Dynamic Range Monocular Vision System for Advanced Driver Assistance Systems

In this paper, we propose a monocular multiframe high dynamic range (HDR) monocular vision system to improve the imaging quality of traditional CMOS/charge-coupled device (CCD)-based vision system for advanced driver assistance systems (ADASs). Conventional CMOS/CCD image sensors are confined to limited dynamic range that it impairs the imaging quality under undesirable environments for ADAS (e.g., strong contrast of bright and darkness, strong sunlight, headlights at night, and so on). Contrary to current HDR video solutions relying on expensive specially designed sensors, we implement a multiframe HDR algorithm to enable one common CMOS/CCD sensor capturing HDR video. Key parts of the realized HDR vision system are: 1) circular exposure control; 2) latent image calculation; and 3) exposure fusion. We have successfully realized a prototype of monocular HDR vision system and mounted it on our SetCar platform. The effectiveness of this technique is demonstrated by our experimental results, while its bottleneck is the processing time. By exploring the capability of the proposed method in the future, a low-cost HDR vision system can be achieved for ADAS.

I. Introduction

During the past decades, advanced driver assistance systems (ADAS) [1] have made great progresses in both research society and automobile industry. ADAS are systems developed to automate/enhance vehicle systems for safety and better driving based upon vision/camera systems, sensor technology and vehicle communication systems. Usually, CMOS/CCD based vision systems are viewed as essential components of ADAS. ADAS applications utilize image sensors to provide enhanced safety features such as parking assistance, lane departure warnings and collision avoidance systems. However, performances of cameras degrade in wide dynamic range scenarios (e.g. low-angle strong sunlight, the headlights of oncoming vehicles, shadows in summer), because in these situations, dynamic range exceeds the capabilities of the conventional CMOS/CCD image sensors. Dynamic range is the ratio of the highest (lightest) signal which an imaging sensor can record to the lowest (darkest) signal. The dynamic range of the real world ranges from (starlight) up to (direct sunlight). A typical nature scene has a contrast ratio around 10,000:1. Common CCD sensors accumulate charge in a “potential well” that is proportional to the number of photons that struck the sensor in that pixel. The size and depth of the “potential well” determines the dynamic range capability of the sensor. Usually, a CMOS/CCD sensor can acquire a contrast of roughly 1,000:1 (60 dB) dynamic range of intensities. The darkest signal is constrained by the thermal noise, or “darkcurrent,” of the sensor. The brightest signal is limited by the total amount of charge that can be accumulated in a single pixel. Image sensors are built such that this total maximum charge is greater than 1,000 times the charge generated thermally. This implies that if the scene has a higher dynamic range, the sensor will not be able to capture it, and the resulting image always has saturated regions in the highlights and underexposed regions in the shadows (as shown in Fig. 1). The degeneration of imaging quality in wide dynamic range scenarios injures camera based ADAS performance [2]. Unfortunately, WDR scenarios appear frequently in real road environments (imaging you drive in the shadow caused by the buildings, or drive in the noon of summer) that make this issue critical for developing better ADAS in the future. For instance, when a car reverses into a garage, the inside of the garage is hard to see because of the contrast between the dark interior and bright daylight. Fig. 1.

Image quality deteriorates in wide dynamic range (WDR) environment.