Object-Driven Multi-Layer Scene Decomposition From a Single Image

We present a method that tackles the challenge of predicting color and depth behind the visible content of an image. Our approach aims at building up a Layered Depth Image (LDI) from a single RGB input, which is an efficient representation that arranges the scene in layers, including originally occluded regions. Unlike previous work, we enable an adaptive scheme for the number of layers and incorporate semantic encoding for better hallucination of partly occluded objects. Additionally, our approach is object-driven, which especially boosts the accuracy for the occluded intermediate objects. The framework consists of two steps. First, we individually complete each object in terms of color and depth, while estimating the scene layout. Second, we rebuild the scene based on the regressed layers and enforce the recomposed image to resemble the structure of the original input. The learned representation enables various applications, such as 3D photography and diminished reality, all from a single RGB image.

1. Introduction

Completing a scene beyond the partial occlusion of its components is a strongly desired property for many computer vision applications. For instance, in robotic manipulation, the ability to see the full target object despite the presence of occluding elements can lead to a more successful and precise grasping. In the autonomous driving context the estimation of the full profile and location of potential obstacles occluded by the car in front of us would prove useful to increase the robustness of the trajectory planning and safety control.