Real-Time Neural Rasterization for Large Scenes

We propose a new method for realistic real-time novel-view synthesis (NVS) of large scenes. Existing neural rendering methods generate realistic results, but primarily work for small scale scenes (< 50m²) and have difficulty at large scale (> 10000m²). Traditional graphics-based rasterization rendering is fast for large scenes but lacks realism and requires expensive manually created assets. Our approach combines the best of both worlds by taking a moderate-quality scaffold mesh as input and learning a neural texture field and shader to model view-dependant effects to enhance realism, while still using the standard graphics pipeline for real-time rendering. Our method outperforms existing neural rendering methods, providing at least 30× faster rendering with comparable or better realism for large self-driving and drone scenes. Our work is the first to enable real-time rendering of large real-world scenes.

1. Introduction

Neural Scene Rasterization. Our method renders urban driving scenes (1920×1080) at high quality and >100 FPS by leveraging neural textures and fast rasterization. We reconstruct driving scenes in Bay Area and show the rendering at four streets on the map. Please refer to our project page https://waabi.ai/neuras/ for more results on driving scenes as well as drone scenes.