SRL-TR2: A Safe Reinforcement Learning Based TRajectory TRacker Framework

This paper aims to solve the trajectory tracking control problem for an autonomous vehicle based on reinforcement learning methods. Existing reinforcement learning approaches have found limited successful applications on safety-critical tasks in the real world mainly due to two challenges: 1) sim-to-real transfer; 2) closed-loop stability and safety concern. In this paper, we propose an actor-critic-style framework SRL-TR2, in which the RL-based TRajectory TRackers are trained under the safety constraints and then deployed to a full-size vehicle as the lateral controller. To improve the generalization ability, we adopt a light-weight adapter State and Action Space Alignment (SASA) to establish mapping relations between the simulation and reality. To address the safety concern, we leverage an expert strategy to take over the control when the safety constraints are not satisfied. Hence, we conduct safe explorations during the training process and improve the stability of the policy. The experiments show that our agents can achieve one-shot transfer across simulation scenarios and unseen realistic scenarios, finishing the field tests with average running time less than 10 ms/step and average lateral error less than 0.1 m under the speed ranging from 12 km/h to 18 km/h. A video of the field tests is available at https://youtu.be/pjWcN_fV24g.

I. Introduction

Autonomous driving has drawn great attention from both academia and industry in recent decades due to its potential to thoroughly change people’s mobility and transport [1], [2]. The key technologies of autonomous driving include environment perception, decision making, path planning, and trajectory tracking control [3]. Among them, trajectory tracking control is an essential and challenging module which ensures passengers’ safety as well as driving experience.