Robust Planar Surface Extraction from Noisy and Semi-Dense 3D Point Cloud for Augmented Reality

The rapid development of augmented reality (AR) has raised various applications nowadays, and it has also attracted great attentions from researchers. Among all the related research topics, real-time 3D reconstruction of real scene, virtual object registration and blending with real scene are the most fundamental ones. However, most real time single camera-based methods such as Simultaneous Localization and Mapping (SLAM) can only output sparse/semi-dense 3D point cloud of the real scene, which is usually too noisy and inaccurate to help virtual object registration and blending. To solve this problem, this paper proposes extracting planar surfaces from noisy and semi-dense 3D point cloud, and uses them to guide virtual object registration and blending. To this end, our method first implements a standard SLAM method to produce the initial 3D point cloud from multiple frames. Then, for each frame we extract line segment candidates and back-project them to the point cloud. By introducing a line's multi-view 3D consistency constraint, the 2D and 3D line segments are jointly optimized to generate reliable 3D line segments. Finally, planar surfaces are obtained from these 3D lines with the coplanar constraints. We conduct various experiments using both indoor and outdoor scenes, and the results show that virtual objects can be blended into real scenes with good geometrical accuracy and satisfying visual quality.