Simulating City-Scale Aerial Data Collection Using Unreal Engine

The rendering quality of game engines has significantly improved in realism. Physically-based rendering techniques started to overtake traditional rasterization-based graphics rendering pipelines in interactive rendering applications. We leverage these advancements to replicate our real-world city-scale aerial image data collections in the Unreal Engine. The use of city-scale synthetic data has many benefits in computer vision and remote sensing research. We can generate ground truth data for many applications, such as feature extraction, bundle adjustment, 3D reconstruction, object tracking, shadow detection, and shadow removal. This is crucial since sufficient ground truth data is not always publicly available for these use cases. In this paper, we introduce an easy-to-use Unreal Engine Blueprint pipeline that replicates drone and plane flight data collections in a circular orbit. Then, we highlight the issues arising from the approximate algorithms used by game engines as proxies for real-world data, and offer solutions to these challenges. Finally, we present our use cases, showing how we use the synthetic data generated by this approach in various research applications.

I. Introduction

Wide Area Motion Imagery (WAMI) datasets are helpful in many geospatial research applications. We have been developing feature extraction, bundle adjustment, 3D reconstruction, shadow detection, orthorectification, georegistration, and object tracking algorithms using WAMI datasets provided by Transparent Sky [1]. Transparent Sky data collections include a number of cities, such as Albuquerque, Berkeley, Columbia, Ferguson, Los Angeles, St. Louis, San Francisco, Syracuse, and more. The images are captured by a crewed aircraft flying in a circular track above the city at around 2 kilometers of altitude. It is a costly and challenging process to collect all this data. The images cover large city-scale environments, so producing manual ground truth for machine learning applications is not feasible, and automatically generated ground truths may not be precise enough. Also, the variations in weather conditions can be captured in the images are limited. Therefore, using Unreal Engine [2], we started simulating cityscale aerial data collections in a synthetic environment.