High-Resolution Shape Completion Using Deep Neural Networks for Global Structure and Local Geometry Inference

We propose a data-driven method for recovering missing parts of 3D shapes. Our method is based on a new deep learning architecture consisting of two sub-networks: a global structure inference network and a local geometry refinement network. The global structure inference network incorporates a long short-term memorized context fusion module (LSTM-CF) that infers the global structure of the shape based on multi-view depth information provided as part of the input. It also includes a 3D fully convolutional (3DFCN) module that further enriches the global structure representation according to volumetric information in the input. Under the guidance of the global structure network, the local geometry refinement network takes as input local 3D patches around missing regions, and progressively produces a high-resolution, complete surface through a volumetric encoder-decoder architecture. Our method jointly trains the global structure inference and local geometry refinement networks in an end-to-end manner. We perform qualitative and quantitative evaluations on six object categories, demonstrating that our method outperforms existing state-of-the-art work on shape completion.

1. Introduction

Inferring geometric information for missing regions of 3D shapes is a fundamental problem in the fields of computer vision, graphics and robotics. With the increasing availability of consumer depth cameras and geometry acquisition devices, robust reconstruction of complete 3D shapes from noisy, partial geometric data remains a challenging problem. In particular, a significant complication is the existence of large missing regions in the acquired 3D data due to occlusions, reflective material properties, and insufficient lighting conditions. Traditional geometry-based methods, such as Poisson surface reconstruction ([12]), are only able to handle relatively small gaps in the acquired 3D data. Unfortunately, these methods often fail to repair large missing regions. Learning-based approaches are more suitable for this task because of their ability to learn powerful 3D shape priors from large online 3D model collections (e.g., ShapeNet, Trimble Warehouse) for repairing such missing regions.