Seamless Scene Segmentation

In this work we introduce a novel, CNN-based architecture that can be trained end-to-end to deliver seamless scene segmentation results. Our goal is to predict consistent semantic segmentation and detection results by means of a panoptic output format, going beyond the simple combination of independently trained segmentation and detection models. The proposed architecture takes advantage of a novel segmentation head that seamlessly integrates multi-scale features generated by a Feature Pyramid Network with contextual information conveyed by a light-weight DeepLab-like module. As additional contribution we review the panoptic metric and propose an alternative that overcomes its limitations when evaluating non-instance categories. Our proposed network architecture yields state-of-the-art results on three challenging street-level datasets, i.e. Cityscapes, Indian Driving Dataset and Mapillary Vistas.

1. Introduction

Scene understanding is one of the grand goals for automated perception that requires advanced visual comprehension of tasks like semantic segmentation (Which semantic category does a pixel belong toƒ) and detection or instance-specific semantic segmentation (Which individual object segmentation mask does a pixel belong toƒ). Solving these tasks has large impact on a number of applications, including autonomous driving or augmented reality. Interestingly, and despite sharing some obvious commonalities, both these segmentation tasks have been predominantly handled in a disjoint way ever since the rise of deep learning, while earlier works [49], [50], [56] already approached them in a joint manner. Instead, independent trainings of models, with separate evaluations using corresponding performance metrics, and final fusion in a post-processing step based on task-specific heuristics have seen a revival.