Networked and Multimodal 3D Modeling of Cities for Collaborative Virtual Environments

3D city-scale models are useful in a number of applications, including education, city planning, navigation systems, artificial intelligence training, and simulations. However, final models need to be immersive and interactive, which requires a mixed reality (XR) environment design that combines e.g., a Cave Automatic Virtual Environment (CAVE) VR system with the Microsoft Hololens2 in a networked and multimodal setting. In this paper, we propose a pipeline to convert a city-scale point cloud into a finalized city-scale textured mesh in which, a number of XR devices can share the same environment and co-exist in a shared space for model interactions. Specifically, we use input point clouds obtained from wide area motion imagery systems or off-the-shelf drones pertaining to Albuquerque, New Mexico, but the pipeline is generalized so that other input can be used. Using four different traditional algorithms and an additional deep learning method, we create meshes for the model interactions. For each mesh produced, we map high-resolution textures onto them, producing a more accurate city, which is then passed into the shared/networked Unity environment. Ten participants provided their assessment of mesh quality and interactivity of the networked environment during exploration of different city reconstructions with the CAVE and laptop device modalities. Results on the perceptual immersive quality of the Point2Mesh deep learning meshes highlights the need for improvements to handle large city scale point clouds.

I. Introduction

Because of the rise in popularity and utilization of mixed reality (XR) devices in everyday life, there is a need to explore all of the possible applications. While there has been some research into the application of city visualization, such as Davis et al. [1], there is room for improvement in the meshing algorithms and the texturing process. The meshing algorithms are what turn a 3D point cloud, or a large set of points in 3D space, into a mesh that is more recognizable and usable. The texturing process applies the real life texture to this mesh, adding realism to the mesh system.