Efficiently selecting regions for scene understanding

Recent advances in scene understanding and related tasks have highlighted the importance of using regions to reason about high-level scene structure. Typically, the regions are selected beforehand and then an energy function is defined over them. This two step process suffers from the following deficiencies: (i) the regions may not match the boundaries of the scene entities, thereby introducing errors; and (ii) as the regions are obtained without any knowledge of the energy function, they may not be suitable for the task at hand. We address these problems by designing an efficient approach for obtaining the best set of regions in terms of the energy function itself. Each iteration of our algorithm selects regions from a large dictionary by solving an accurate linear programming relaxation via dual decomposition. The dictionary of regions is constructed by merging and intersecting segments obtained from multiple bottom-up over-segmentations. To demonstrate the usefulness of our algorithm, we consider the task of scene segmentation and show significant improvements over state of the art methods.

1. Introduction

As low-level vision reaches maturity, more and more researchers have started considering high-level scene understanding tasks such as scene segmentation, object detection and single view 3D reconstruction. Given the success of pixel-based approaches for low-level vision (such as image denoising, where each pixel is assigned an intensity value; or stereo reconstruction, where each pixel is assigned a disparity value), the initial expectation was that such models will also be suitable for scene understanding. However, this expectation has turned out to be overly optimistic as features extracted from patches surrounding pixels are prone to noise from background clutter. To address this issue, many researchers have advocated the use of regions, sets of connected pixels that share common characteristics such as color or texture, to extract reliable discriminative features.