CGA-MAE: Camera-Guided Alignment to Auxiliary Point Cloud Masked Autoencoders for 3D Object Detection

Masked Autoencoders have demonstrated remarkable proficiency in acquiring robust visual representations and achieving cutting-edge outcomes across diverse independent modalities, However, only a handful of studies have delved into their efficacy within multi-modality environments. In this work, we focus on LiDAR point cloud and Camera image, two modalities commonly coexistent in real-world scenarios, aiming to explore their intricate interactions. To improve upon the cross-modal synergy in existing works, we project the feature in the bird's eye view(BEV) plane to receive more geometry information and propose CGA-MAE, a simple and effective cross-modal fusion self-supervised pretraining method based on Camera-Guided Alignment (CGA), which can bridge the BEV feature learning gap between LiDAR and Camera. We initially recognize the significance of masking strategies between these sources, leveraging a projection module, we harmonize the alignment of masks and visible tokens inherent in both modalities. Then, we propose a new Modality Feature Fusion Model called MFFM to promote the interaction between the LiDAR feature and the Camera Feature. Through extensive experiments performed on large NuScenes Dataset, we discover it is nontrivial to interactively learn LiDAR-Camera features, where Our approach significantly advances the State-Of-The-Art (SOTA), achieving a mean Average Precision (mAP) of 67.8% and a nuScenes Detection Score (NDS) of 71.0%.