Cross-modal Matching CNN for Autonomous Driving Sensor Data Monitoring

Multiple sensor types have been increasingly used in modern autonomous driving systems (ADS) to ensure safer perception. Through applications of multiple modalities of perception sensors that differ in their physical properties, obtained data complement to each other and provide a more robust view of surroundings. On the other hand, however, sensor data fault is inevitable thus lead to wrong perception results and consequently endangers the overall safety of the vehicle. In this paper, we present a cross-modal Convolutional Neural Networks (CNN) for autonomous driving sensor data monitoring functions, such as fault detection and online data quality assessment. Assuming the overlapping view of different sensors should be consistent under normal circumstances, we detect anomalies such as missynchronisation through matching camera image and LIDAR point cloud. A masked pixel-wise metric learning loss is proposed to improve exploration of the local structures and to build an alignment-sensitive pixel embedding. In our experiments with a selected KITTI dataset and specially tailored fault data generation methods, the approach shows a promising success for sensor fault detection and point cloud quality assessment (PCQA) results.