Multimodal Object Detection and Ranging Based on Camera and Lidar Sensor Fusion for Autonomous Driving

A robust perception system is critical in autonomous driving. It is responsible for object detection, classification, and ranging under challenging circumstances. Camera and lidar sensors provide complementary information, and by combining these two modalities, we can increase the robustness and accuracy of the overall perception system. This paper presents the implementation of sensor fusion based perception using camera images and lidar point clouds for object detection and ranging in a real-time driving environment. The experiment results obtained with our test vehicle demonstrate that the perception of vehicle surroundings can be more effectively achieved by means of camera-lidar sensor fusion compared with using a single type of sensor.

I. Introduction

Object detection by a single type of sensor is not sufficient to ensure safety for automated driving systems. Every sensor has its advantages and shortcomings. Identification of color, shape, and type of obstacles is relatively easy with cameras compared with other sensors. But generally, cameras cannot capture depth or range information. Methods are available to recover 3D information from camera images. However, this is the fundamental problem with the cameras. In contrast, lidars can depict the 3D surroundings of the vehicle more accurately, often with a much larger field of view.