A Vision-Dynamics Learning Approach to Prediction-Based Control in Autonomous Vehicles

The performance of autonomous driving systems tends to be particularly high when the dynamics of the surrounding objects are tracked and predicted in advance. Although there is significant work in the area of object tracking and prediction, there is less in prediction of raw sensory information. In this paper, we propose a vision-dynamics approach to prediction-based control of autonomous vehicles, coined ObserveNet. Given historical sensing data, the method aims to learn the dynamics of the observations in a self-supervised manner, with the purpose of estimating an autonomous vehicle's future observations. We use ObserveNet to predict the future sensory observations of the ego vehicle, which are then fed to a local path planner. The estimator is encoded within a deep neural network, taking as input historical observations and vehicle states: position, velocity and heading. The first round of experiments is performed in the CARLA simulator, evaluating the capabilities of the path planner in aggressive driving contexts (e.g. overtaking maneuvers, side cut-off situations, etc.) by comparing it side by side with a classical path planner-controller pipeline. We have also tested the predictions of ObserveNet on a real 1:8 scaled model car.