Geometric Loss Functions for Camera Pose Regression with Deep Learning

Deep learning has shown to be effective for robust and real-time monocular image relocalisation. In particular, PoseNet [22] is a deep convolutional neural network which learns to regress the 6-DOF camera pose from a single image. It learns to localize using high level features and is robust to difficult lighting, motion blur and unknown camera intrinsics, where point based SIFT registration fails. However, it was trained using a naive loss function, with hyper-parameters which require expensive tuning. In this paper, we give the problem a more fundamental theoretical treatment. We explore a number of novel loss functions for learning camera pose which are based on geometry and scene reprojection error. Additionally we show how to automatically learn an optimal weighting to simultaneously regress position and orientation. By leveraging geometry, we demonstrate that our technique significantly improves PoseNets performance across datasets ranging from indoor rooms to a small city.

1. Introduction

Designing a system for reliable large scale localisation is a challenging problem. The discovery of the positioning system in mammalian brains, located in the hippocampus, was awarded the 2014 Nobel prize in Physiology or Medicine [36], [32]. It is an important problem for computer vision too, with localisation technology essential for many applications including autonomous vehicles, unmanned aerial vehicles and augmented reality. State of the art localisation systems perform very well within controlled environments [24], [34], [12], [33], [44]. However, we are yet to see their wide spread use in the wild because of their inability to cope with large viewpoint or appearance changes.