Dynamic Fluid Surface Reconstruction Using Deep Neural Network

Recovering the dynamic fluid surface is a long-standing challenging problem in computer vision. Most existing image-based methods require multiple views or a dedicated imaging system. Here we present a learning-based single-image approach for 3D fluid surface reconstruction. Specifically, we design a deep neural network that estimates the depth and normal maps of a fluid surface by analyzing the refractive distortion of a reference background image. Due to the dynamic nature of fluid surfaces, our network uses recurrent layers that carry temporal information from previous frames to achieve spatio-temporally consistent reconstruction given a video input. Due to the lack of fluid data, we synthesize a large fluid dataset using physics-based fluid modeling and rendering techniques for network training and validation. Through experiments on simulated and real captured fluid images, we demonstrate that our proposed deep neural network trained on our fluid dataset can recover dynamic 3D fluid surfaces with high accuracy.

1. Introduction

Dynamic fluid phenomena are common in our environment. Accurate 3D reconstruction of the fluid surface helps advance our understanding of the presence and dynamics of the fluid phenomena and thus benefits many scientific and engineering fields ranging from hydraulics and hydrodynamics [5], [20] to 3D animation and visualization [13]. However, it is difficult to tackle this problem with nonintrusive image-based methods as the captured images are often severely distorted by the refraction of light that happens at the fluid-air interface. This is because to extract invariant and reliable image features under distortion is highly challenging. Further, the dynamic nature of fluid flow makes this problem even more challenging as we need to recover a sequence of 3D surfaces that are consistent both spatially and temporally to represent the fluid motion.