PlaneRCNN: 3D Plane Detection and Reconstruction From a Single Image

This paper proposes a deep neural architecture, PlaneRCNN, that detects and reconstructs piecewise planar regions from a single RGB image. PlaneRCNN employs a variant of Mask R-CNN to detect planes with their plane parameters and segmentation masks. PlaneRCNN then refines an arbitrary number of segmentation masks with a novel loss enforcing the consistency with a nearby view during training. The paper also presents a new benchmark with more fine-grained plane segmentations in the ground-truth, in which, PlaneRCNN outperforms existing state-of-the-art methods with significant margins in the plane detection, segmentation, and reconstruction metrics. PlaneRCNN makes an important step towards robust plane extraction method, which would have immediate impact on a wide range of applications including Robotics, Augmented Reality, and Virtual Reality.

1. Introduction

Planar regions in 3D scenes offer important geometric cues in a variety of 3D perception tasks such as scene understanding [42], scene reconstruction [3], and robot navigation [18], [56]. Accordingly, piecewise planar scene reconstruction has been a focus of computer vision research for many years, for example, plausible recovery of planar structures from a single image [16], volumetric piecewise planar reconstruction from point clouds [3], and Manhattan depthmap reconstruction from multiple images [11].