Deep Depth Estimation from Thermal Image

Robust and accurate geometric understanding against adverse weather conditions is one top prioritized conditions to achieve a high-level autonomy of self-driving cars. However, autonomous driving algorithms relying on the visible spectrum band are easily impacted by weather and lighting conditions. A long-wave infrared camera, also known as a thermal imaging camera, is a potential rescue to achieve high-level robustness. However, the missing necessities are the well-established large-scale dataset and public benchmark results. To this end, in this paper, we first built a large-scale Multi-Spectral Stereo (MS²) dataset, including stereo RGB, stereo NIR, stereo thermal, and stereo LiDAR data along with GNSS/IMU information. The collected dataset provides about 195K synchronized data pairs taken from city, residential, road, campus, and suburban areas in the morning, daytime, and nighttime under clear-sky, cloudy, and rainy conditions. Secondly, we conduct an exhaustive validation process of monocular and stereo depth estimation algorithms designed on visible spectrum bands to benchmark their performance in the thermal image domain. Lastly, we propose a unified depth network that effectively bridges monocular depth and stereo depth tasks from a conditional random field approach perspective. Our dataset and source code are available at https://github.com/UkcheolShin/MS2-MultiSpectralStereoDataset.

1. Introduction

Recently, a number of researches have been conducted for accurate and robust geometric understanding in self-driving cars based on the widely-used benchmark datasets, such as KITTI [15], DDAD [17], and nuScenes [4]. Modern computer vision algorithms deploy a deep neural network and data-driven machine learning technique to achieve high-level accuracy, which needs large-scale datasets. However, from the perspective of robustness in real-world, the algorithms mostly rely on visible spectrum images and are easily degenerated by weather and lighting conditions. Figure 1.

Depth from thermal images in various environments. our proposed network can estimate both monocular and stereo depth maps regardless of given a single or stereo thermal image via unified network architecture. furthermore, depth estimation results from thermal images show high-level reliability and robustness under day-light, low-light, and rainy conditions.