Neural Brdf Plugin for Unsupervised Photometric Stereo

Recovering surface normal under unknown general reflectance is challenging for photometric stereo. Supervised learning-based methods implicitly handle general reflectances by learning from training data. However, it suffers from a demand for a large number of rendered images as training data and also from a domain gap due to the biased shape distribution in the training set. Unsupervised methods mitigate these issues by assessing image re-rendering losses, but their performance is limited primarily due to the fact that multiple combinations of reflectance parameters can explain the same appearance. We combine the merits of these two approaches by introducing a BRDF plugin, a neural network-based BRDF dictionary that is trained using a measured BRDF dataset, offering a data prior to effectively constrain reflectances. Specifically, it maps each measured BRDF to a unique latent code, forming a latent BRDF space that serves as the data prior for reflectances. Building upon the BRDF plugin, we develop a new unsupervised photometric stereo method to recover both surface normal and reflectance, which exhibits superior accuracy than previous methods in experiments on synthetic and real-world datasets
^1
1
Project done in Osaka University..