Fusing LiDAR and Color Imagery for Object Detection using Convolutional Neural Networks

The goal of this paper is answer to this question: how much fusing LiDAR and color images can improve the performance of a convolutional neural network (CNN)-based detector? To this end, we trained state-of-the-art CNN-based detectors using different configurations of color images and their associated LiDAR data, in conjunction and independently. Moreover, we investigate the effect of sparse and dense LiDAR data on the detection accuracy. For this purpose, we estimate a dense depth image from spare LiDAR data using a recent self-supervised depth completion technique [1] that requires only sequences of color and sparse depth images, without the need for dense depth labels. Then, we compared two detectors when are trained on sparse or dense LiDAR data. The obtained results on the KITTI dataset show that fusing dense LiDAR and color images is an efficient solution for future object detectors.