Virtual IR Sensing for Planetary Rovers: Improved Terrain Classification and Thermal Inertia Estimation

Terrain classification is critically important for Mars rovers, which rely on it for planning and autonomous navigation. On-board terrain classification using visual information has limitations, and is sensitive to illumination conditions. Classification can be improved if one fuses visual imagery with additional infrared (IR) imagery of the scene, yet unfortunately there are no IR image sensors on the current Mars rovers. A virtual IR sensor, estimating IR from RGB imagery using deep learning, was proposed in the context of a MU-Net architecture. However, virtual IR estimation was limited by the fact that slope angle variations induce temperature differences within the same terrain. This paper removes this limitation, giving good IR estimates and as a consequence improving terrain classification by including the additional angle from the surface normal to the Sun and the measurement of solar radiation. The estimates are also useful when estimating thermal inertia, which can enhance slip prediction and small rock density estimation. Our approach is demonstrated in two applications. We collected a new data set to verify the effectiveness of the proposed approach and show its benefit by applying to the two applications.