3D Bounding Box Estimation Using Deep Learning and Geometry

We present a method for 3D object detection and pose estimation from a single image. In contrast to current techniques that only regress the 3D orientation of an object, our method first regresses relatively stable 3D object properties using a deep convolutional neural network and then combines these estimates with geometric constraints provided by a 2D object bounding box to produce a complete 3D bounding box. The first network output estimates the 3D object orientation using a novel hybrid discrete-continuous loss, which significantly outperforms the L2 loss. The second output regresses the 3D object dimensions, which have relatively little variance compared to alternatives and can often be predicted for many object types. These estimates, combined with the geometric constraints on translation imposed by the 2D bounding box, enable us to recover a stable and accurate 3D object pose. We evaluate our method on the challenging KITTI object detection benchmark [2] both on the official metric of 3D orientation estimation and also on the accuracy of the obtained 3D bounding boxes. Although conceptually simple, our method outperforms more complex and computationally expensive approaches that leverage semantic segmentation, instance level segmentation and flat ground priors [4] and sub-category detection [23][24]. Our discrete-continuous loss also produces state of the art results for 3D viewpoint estimation on the Pascal 3D+ dataset[26].

1. Introduction

The problem of 3D object detection is of particular importance in robotic applications that require decision making or interactions with objects in the real world. 3D object detection recovers both the 6 DoF pose and the dimensions of an object from an image. While recently developed 2D detection algorithms are capable of handling large variations in viewpoint and clutter, accurate 3D object detection largely remains an open problem despite some promising recent work. The existing efforts to integrate pose estimation with state-of-the-art object detectors focus mostly on viewpoint estimation. They exploit the observation that the appearance of objects changes as a function of viewpoint and that discretization of viewpoints (parametrized by azimuth and elevation) gives rise to sub-categories which can be trained discriminatively [23]. In more restrictive driving scenarios alternatives to full 3D pose estimation explore exhaustive sampling and scoring of all hypotheses [4] using a variety of contextual and semantic cues. Figure 1.

Our method takes the 2D detection bounding box and estimates a 3D bounding box.