Integrating videos with LIDAR scans for virtual reality

LIDAR range scans can be used to quickly create accurate 3D models for virtual reality and as a basis to visualize sets of photographs, videos, and virtual objects in a cohesive environment. The number of existing virtual reality programs that use LIDAR data as input has motivated our group to develop methods for fusing images with 3D scans and for augmenting the scans with both dynamic objects present in videos and virtual models. Bringing together as many data sources as possible increases users' abilities to present related information in one, intuitive venue. We demonstrate how to register a variety of 2D imagery with a range scan to construct photo-realistic models and to extract walking people captured in videos and model them in a 3D space. We also present a method for determining the sun position from a set of stitched photographs in order to apply correct lighting to virtual objects placed amongst real world data. Naturally lit objects can be inserted into original photographs using our 2D-3D registration information. These methods are all combined to display and study photos, videos, and virtual objects in a complete 3D environment.

1 Introduction

A central component of setting up many virtual reality systems is acquiring models and data to display and visualize. Using LIDAR (Light Detection and Ranging) range scans as a basis for generating 3D models has a great number of benefits, including quickly creating realistic representations of scanned objects and easily presenting 2D and 3D information from a variety of perspectives [1]–[4]. We present methods for registering 2D imagery with range scans in order to produce photo-realistic renderings of buildings and foliage and creating a virtual reality system. This system includes walking people captured in videos and static and dynamic virtual objects. The data collection process is quite simple. The user only needs to scan a scene once with a range scanner and take photographs and videos of a scene. With this information we can build a full 3D virtual representation displaying all the photographs and videos at once.