Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset

As autonomous driving systems mature, motion forecasting has received increasing attention as a critical requirement for planning. Of particular importance are interactive situations such as merges, unprotected turns, etc., where predicting individual object motion is not sufficient. Joint predictions of multiple objects are required for effective route planning. There has been a critical need for high-quality motion data that is rich in both interactions and annotation to develop motion planning models. In this work, we introduce the most diverse interactive motion dataset to our knowledge, and provide specific labels for interacting objects suitable for developing joint prediction models. With over 100,000 scenes, each 20 seconds long at 10 Hz, our new dataset contains more than 570 hours of unique data over 1750 km of roadways. It was collected by mining for interesting interactions between vehicles, pedestrians, and cyclists across six cities within the United States. We use a high-accuracy 3D auto-labeling system to generate high quality 3D bounding boxes for each road agent, and provide corresponding high definition 3D maps for each scene. Furthermore, we introduce a new set of metrics that provides a comprehensive evaluation of both single agent and joint agent interaction motion forecasting models. Finally, we provide strong baseline models for individual-agent prediction and joint-prediction. We hope that this new large-scale interactive motion dataset will provide new opportunities for advancing motion forecasting models.

1. Introduction

Motion forecasting has received increasing attention as a critical requirement for planning in autonomous driving systems [8], [14], [40], [36], [28], [34]. Due to the complexity of scenes that autonomous systems need to safely handle, predicting object motion in the scene is a difficult task, suitable for machine learning models. Building effective motion forecasting models requires large amounts of high quality real world data. Creating a dataset for motion forecasting is complicated by the fact that the distribution of real world data is highly imbalanced [4], [18], [32], [38]; in the common case, vehicles drive straight at a constant velocity. In order to develop effective models, a dataset must contain and measure performance on a wide range of behaviors and trajectory shapes for different object types that an autonomous system will encounter in operation.