OmniMVS: End-to-End Learning for Omnidirectional Stereo Matching

In this paper, we propose a novel end-to-end deep neural network model for omnidirectional depth estimation from a wide-baseline multi-view stereo setup. The images captured with ultra wide field-of-view (FOV) cameras on an omnidirectional rig are processed by the feature extraction module, and then the deep feature maps are warped onto the concentric spheres swept through all candidate depths using the calibrated camera parameters. The 3D encoder-decoder block takes the aligned feature volume to produce the omnidirectional depth estimate with regularization on uncertain regions utilizing the global context information. In addition, we present large-scale synthetic datasets for training and testing omnidirectional multi-view stereo algorithms. Our datasets consist of 11K ground-truth depth maps and 45K fisheye images in four orthogonal directions with various objects and environments. Experimental results show that the proposed method generates excellent results in both synthetic and real-world environments, and it outperforms the prior art and the omnidirectional versions of the state-of-the-art conventional stereo algorithms.

1. Introduction

Image-based depth estimation, including stereo and multi-view dense reconstruction, has been widely studied in the computer vision community for decades. In conventional two-view stereo matching, deep learning methods [12, 4] have achieved drastic performance improvement recently. Besides, there are strong needs on omnidirectional or wide FOV depth sensing in autonomous driving and robot navigation to sense the obstacles and surrounding structures. Human drivers watch all directions, notjust the front, and holonomic robots need to sense all directions to move freely. However, conventional stereo rigs and algorithms cannot capture or estimate ultra wide FOV depth maps. Merging depth maps from multiple conventional stereo pairs can be one possibility, but the useful global context information cannot be propagated between the pairs and there might be a discontinuity at the seam.