PRN: Panoptic Refinement Network

Panoptic segmentation is the task of uniquely assigning every pixel in an image to either a semantic label or an individual object instance, generating a coherent and complete scene description. Many current panoptic segmentation methods, however, predict masks of semantic classes and object instances in separate branches, yielding inconsistent predictions. Moreover, because state-of-the-art panoptic segmentation models rely on box proposals, the instance masks predicted are often of low-resolution. To overcome these limitations, we propose the Panoptic Refinement Network (PRN), which takes masks from base panoptic segmentation models and refines them jointly to produce coherent results. PRN extends the offset map-based architecture of Panoptic-Deeplab with several novel ideas including a foreground mask and instance bounding box offsets, as well as coordinate convolutions for improved spatial prediction. Experimental results on COCO and Cityscapes show that PRN can significantly improve already accurate results from a variety of panoptic segmentation networks.

1. Introduction

Panoptic segmentation addresses semantic and instance segmentation in a unified way, aiming to assign each pixel to one of the background classes (i.e. stuff) or one of the object instances (i.e. things) [22]. Facilitated by the introduction of several open-source datasets (e.g. Cityscapes [11], COCO [33], Mapillary Vistas [37]), panoptic segmentation has quickly become a popular research topic leading to significant progress [8], [9, [17], [21], [24], [26], [27], [34], [45], [47], [49], [51] since its introduction.