Integrated Sensor Fusion Based on 4D MIMO Radar and Camera: A Solution for Connected Vehicle Applications

This article presents an integrated sensor fusion (ISF) solution based on the multiple-input, multiple-output (MIMO) radar, camera, and on-device computing. The MIMO radar is capable of estimating an object’s attributes in four dimensions—range, velocity, azimuth angle, and elevation angle—which can be further used to estimate the length, width, and height of the object. The camera is responsible for object classification based on deep learning. The respective signal processing pipelines and the fusion of results are carried by the on-device computing platform. These two sensors complement each other very well in detecting and classifying traffic objects. Compared with existing sensor fusion solutions based on multiple distributed devices, ISF exhibits superior performance in terms of latency and the total cost of ownership (TCO). It also simplifies time synchronization among different sensors and facilitates the deeper fusion of the signal processing algorithms of different sensors. The comprehensive roadside sensing capabilities provided by the ISF solution can enhance the safety and efficiency of both the automated driving and human driving of connected vehicles.