Scene-level Point Cloud Colorization with Semantics-and-geometry-aware Networks

In robotic applications, we often obtain tons of 3D point cloud data without color information, and it is difficult to visualize point clouds in a meaningful and colorful way. Can we colorize 3D point clouds for better visualization? Existing deep learning-based colorization methods usually only take simple 3D objects as input, and their performance for complex scenes with multiple objects is limited. To this end, this paper proposes a novel semantics-and-geometry-aware colorization network, termed SGNet, for vivid scene-level point cloud colorization. Specifically, we propose a novel pipeline that explores geometric and semantic cues from point clouds containing only coordinates for color prediction. We also design two novel losses, including a colorfulness metric loss and a pairwise consistency loss, to constrain model training for genuine colorization. To the best of our knowledge, our work is the first to generate realistic colors for point clouds of large-scale indoor scenes. Extensive experiments on the widely used ScanNet benchmarks demonstrate that the proposed method achieves state-of-the-art performance on point cloud colorization.

I. Introduction

3D sensors (e.g., depth sensor, time-of-flight sensor, and LiDAR) are capable of perceiving fine 3D geometric in-formation of the scene but unable to capture appearance details (e.g., color and texture) of the surroundings, compared with image sensors. In lots of robotic applications, only 3D sensors are utilized without any color information, which makes 3D data visualization challenging. Therefore, it is desirable to visualize with vivid color because colorized 3D data is perceptually more meaningful and credible, which often conveys rich semantics clues, thus not only providing better scene understanding to human beings but also significant improvements for visual recognition [1], [2] in modern AR/VR and robotic applications. As shown in Fig. 1, compared with the original point cloud with coordinates only, with the support of color information, the colorized point cloud makes the scene easier to understand visually, greatly improving the recognizability of objects. Therefore, point cloud colorization is an emerging topic for better 3D data visualization and visual perception.