Finding Things: Image Parsing with Regions and Per-Exemplar Detectors

This paper presents a system for image parsing, or labeling each pixel in an image with its semantic category, aimed at achieving broad coverage across hundreds of object categories, many of them sparsely sampled. The system combines region-level features with per-exemplar sliding window detectors. Per-exemplar detectors are better suited for our parsing task than traditional bounding box detectors: they perform well on classes with little training data and high intra-class variation, and they allow object masks to be transferred into the test image for pixel-level segmentation. The proposed system achieves state-of-the-art accuracy on three challenging datasets, the largest of which contains 45,676 images and 232 labels.

1. Introduction

This paper addresses the problem of image parsing, or labeling each pixel in an image with its semantic category. Our goal is achieving broad coverage - the ability to recognize hundreds or thousands of object classes that commonly occur in everyday street scenes and indoor environments. A major challenge in doing this is posed by the non-uniform statistics of these classes in realistic scene images. A small number of classes - mainly ones associated with large regions or “stuff,” such as road, sky, trees, buildings, etc. - constitute the majority of all image pixels and object instances in the dataset. But a much larger number of “thing” classes - people, cars, dogs, mailboxes, vases, stop signs - occupy a small percentage of image pixels and have relatively few instances each.