A Multimodal 3-D Detector With Attention From the Corresponding Modal

Robust 3-D detection based on camera-light detection and ranging (LiDAR) fusion has become a research focus for autonomous driving. Despite the extensive efforts invested, it remains an extremely challenging task to effectively establish fine-grained corresponding relations between different modalities and achieve complementary multimodal fusion free of interference. To this end, we propose a multimodal 3-D detector termed a corresponding feature-adaptive fusion network (CFFNet), mainly consisting of a multimodal feature extraction backbone network and a corresponding feature-based attention fusion (CFAF) module. Specifically, the multimodal feature extraction backbone is devised to build a simple, yet effective cross-modal correspondence by unifying multimodal features into a shared bird’s eye view (BEV) space. Expressive and noise-suppressed image BEV features are generated from pseudo-image voxels, whose depth information is estimated with range-wise attention drawn from LiDAR BEV features. With the prominent heterogeneity of multimodal features taken into consideration, the CFAF module performs a discriminative fusion targeted at informative and contributive features retained in different modalities. The fusion is accomplished by element-wise reweighting image BEV features with attention generated from differential features of LiDAR. The multimodal feature extraction backbone and CFAF module are both evaluated and proved to offer significant performance gains separately and together. Evaluations conducted on the Karlsruhe Institute of Technology and Toyota Technological Institute (KITTI) 3-D object detection dataset show that the proposed CFFNet achieves state-of-the-art (SOTA) performance.