Learning with Unreliability: Fast Few-Shot Voxel Radiance Fields with Relative Geometric Consistency

We propose a voxel-based optimization framework, Re VoRF, for few-shot radiance fields that strategically ad-dress the unreliability in pseudo novel view synthesis. Our method pivots on the insight that relative depth relationships within neighboring regions are more reliable than the ab-solute color values in disoccluded areas. Consequently, we devise a bilateral geometric consistency loss that carefully navigates the trade-off between color fidelity and geometric accuracy in the context of depth consistency for uncertain regions. Moreover, we present a reliability-guided learning strategy to discern and utilize the variable quality across syn-thesized views, complemented by a reliability-aware voxel smoothing algorithm that smoothens the transition between reliable and unreliable data patches. Our approach allows for a more nuanced use of all available data, promoting en-hanced learning from regions previously considered unsuit-able for high-quality reconstruction. Extensive experiments across diverse datasets reveal that our approach attains significant gains in efficiency and accuracy, delivering ren-dering speeds of 3 FPS, 7 mins to train a 360° scene, and a 5% improvement in PSNR over existing few-shot methods. Code is available at https://github.com/HKCLynn/ReVoRF.

1. Introduction

Neural Radiance Fields (NeRF) have revolutionized the fields of novel view synthesis and 3D reconstruction by lever-aging an implicit function optimized from a collection of 2D images [2], [13], [24],[2]7. Despite their remarkable ren-dering capabilities, N eRFs are hampered by the substantial cost and time required to gather dense image sets for a given scene [6], [41], [47]. This challenge has spurred the development of Few-shot NeRF, an emerging domain focused on reconstructing 3D scenes with minimal image data [6], [8], [12],[18]. Figure 1.

We present Re VoRF, a voxel-based framework designed to capitalize on the unreliability inherent in warped novel views. (b) demonstrates the warping outcomes, where black holes signify unmatched pixels from the original view. (c) illustrates the results of training when these holes are masked out, which unfortunately results in ambiguous geometric structures. In contrast, (d) showcases our approach's ability to maintain correct geometric consistency. Re VoRF achieves this by leveraging relational depth prior know ledge within these unreliable hole regions. Our approach demonstrates the best reconstruction quality while being one of the fastest few-shot approaches in (e).