Unsupervised Surface-to-Orbit View Generation of Planetary Terrain

Newer generations of autonomous spacecraft are utilizing modern machine-learning approaches with labeled data (also known as supervised learning) for visual perception. As labeled data instances are extremely limited or completely nonexistent for applications in the space domain, training data is often acquired through indirect means such as simulation or photorealistic image generation. Alternatively, a powerful yet mostly unexplored approach is to perform geometric transformations over existing data while retaining the labels that result in a format suitable for training a new application. One such application is hazard detection for planetary landings, in which images captured on the surface of a planet by previous spacecraft (e.g. Mars rovers) can be transformed into a representative bird’s-eye (i.e. orbital-style) view through an Inverse Perspective Mapping (IPM). However, this results in the unnatural stretching of pixels further away from the viewport as physical properties in the 3D scene are represented by less pixel information at a distance. To alleviate such issues, we present a generative approach to orbital-style view synthesis that improves the visual fidelity of IPMs on planetary surface terrain. In particular, we describe how to condition generative model learning on input signals given only by surface and IPM images permitting an entirely unsupervised training approach. Furthermore, we show how such conditioning creates images that are consistent in both feature structure and location, allowing for the mapping of auxiliary information like semantic pixel labels of the surface to the synthesized views. Through in-depth qualitative and quantitative analysis, we demonstrate the ability of our method to create less-deformed, more realistic images that directly improve downstream learning tasks.