UniTR: A Unified and Efficient Multi-Modal Transformer for Bird’s-Eye-View Representation

Jointly processing information from multiple sensors is crucial to achieving accurate and robust perception for reliable autonomous driving systems. However, current 3D perception research follows a modality-specific paradigm, leading to additional computation overheads and inefficient collaboration between different sensor data. In this paper, we present an efficient multi-modal backbone for outdoor 3D perception named UniTR, which processes a variety of modalities with unified modeling and shared parameters. Unlike previous works, UniTR introduces a modality-agnostic transformer encoder to handle these view-discrepant sensor data for parallel modal-wise representation learning and automatic cross-modal interaction without additional fusion steps. More importantly, to make full use of these complementary sensor types, we present a novel multi-modal integration strategy by both considering semantic-abundant 2D perspective and geometry-aware 3D sparse neighborhood relations. UniTR is also a fundamentally task-agnostic backbone that naturally supports different 3D perception tasks. It sets a new state-of-the-art performance on the nuScenes benchmark, achieving +1.1 NDS higher for 3D object detection and +12.0 higher mIoU for BEV map segmentation with lower inference latency. Code will be available at https://github.com/Haiyang-W/UniTR.

1. Introduction

Perceiving the physical world in 3D space is critical for reliable autonomous driving systems [2], [54]. As self-driving sensors become more advanced, integrating the complementary signals captured from different sensors (e.g., Cameras, LiDAR, and Radar) in a unified manner is essential. To achieve this goal, we propose UniTR, a unified yet efficient multi-modal transformer backbone that can process both 3D sparse point clouds and 2D multi-view dense images in parallel to learn the unified bird’s-eye-view (BEV) representations for boosting 3D outdoor perception.