A Parametric Top-View Representation of Complex Road Scenes

In this paper, we address the problem of inferring the layout of complex road scenes given a single camera as input. To achieve that, we first propose a novel parameterized model of road layouts in a top-view representation, which is not only intuitive for human visualization but also provides an interpretable interface for higher-level decision making. Moreover, the design of our top-view scene model allows for efficient sampling and thus generation of large-scale simulated data, which we leverage to train a deep neural network to infer our scene model's parameters. Specifically, our proposed training procedure uses supervised domain-adaptation techniques to incorporate both simulated as well as manually annotated data. Finally, we design a Conditional Random Field (CRF) that enforces coherent predictions for a single frame and encourages temporal smoothness among video frames. Experiments on two public data sets show that: (1) Our parametric top-view model is representative enough to describe complex road scenes; (2) The proposed method outperforms baselines trained on manually-annotated or simulated data only, thus getting the best of both; (3) Our CRF is able to generate temporally smoothed while semantically meaningful results.

1. Introduction

Understanding complex layouts of the 3D world is a crucial ability for applications like robot navigation, driver assistance systems or autonomous driving. Recent success in deep learning-based perception systems enables pixel-accurate semantic segmentation [3], [4], [33] and (monocular) depth estimation [9], [15], [32] in the perspective view of the scene. Other works like [10], [23], [25] go further and reason about occlusions and build better representations for 3D scene understanding. The representation in these works, however, is typically non-parametric, i.e., it provides a semantic label for a 2D/3D point of the scene, which makes higher-level reasoning hard for downstream applications.