Stereo Camera Multi-Perception System for Self-Driving Vehicles

Stereo cameras play a crucial role in self-driving vehicles as they provide cost-effective depth estimation. They serve various purposes, including object classification, semantic segmentation, depth estimation, and more. These cameras are valuable sensors for enabling advanced perception capabilities in self-driving vehicle systems. In this work, we proposed multi-perception systems such as object distance measurement, object classification, and object position estimation using stereo camera view. This system utilizes deep neural networks to perform object classification on image frames captured by the stereo camera. Utilizing the calculated disparity image derived from two stereo images obtained from the stereo camera, the system estimates the depth of the detected objects on the vehicle simultaneously. We estimate the position of the object from the y coordinate of the centroid object that came across it, which will be converted to world coordinates. Based on the results, the integration of multi-perception system (distance measurement, object position estimation, and object classification) demonstrated an average accuracy of 97.11% when tested with diverse objects. The system demonstrated an average processing time of 84 milliseconds, or 11.8 frames per second (FPS). Operating at 1.5 GHz on the Jetson Nano 4GB. Furthermore, its capabilities enable multi-sensor fusion for localization, mapping, and path planning in autonomous vehicle applications. These results highlight the system’s potential for enhancing the perception capabilities of self-driving vehicles.