Pixel Consensus Voting for Panoptic Segmentation

The core of our approach, Pixel Consensus Voting, is a framework for instance segmentation based on the generalized Hough transform. Pixels cast discretized, probabilistic votes for the likely regions that contain instance centroids. At the detected peaks that emerge in the voting heatmap, backprojection is applied to collect pixels and produce instance masks. Unlike a sliding window detector that densely enumerates object proposals, our method detects instances as a result of the consensus among pixel-wise votes. We implement vote aggregation and backprojection using native operators of a convolutional neural network. The discretization of centroid voting reduces the training of instance segmentation to pixel labeling, analogous and complementary to FCN-style semantic segmentation, leading to an efficient and unified architecture that jointly models things and stuff. We demonstrate the effectiveness of our pipeline on COCO and Cityscapes Panoptic Segmentation and obtain competitive results. Code will be open-sourced.

1. Introduction

The development of visual recognition algorithms has followed the evolution of recognition benchmarks. PAS-CAL VOC [13] standardizes the task of bounding box object detection and the associated IoU/Average Precision metrics. At the time, the approaches defining the state-of-the-art, DPM [14] and later the R-CNN family [17], [48], address object detection by reasoning about densely enumerated box proposals, following the sliding window classification approach of earlier detectors [52], [51]. SDS [19] expands the scope of object detection to include instance mask segmentation, and introduces early versions of the mAP^bbox and mAP^mask, subsequently popularized by the COCO dataset [35]. Bounding boxes, however, remain the primary vehicle for object reasoning.