Safe Deep Reinforcement Learning for Adaptive Cruise Control by Imposing State-Specific Safe Sets

Deep reinforcement learning has been increasingly discussed for solving continuous control tasks in the field of autonomous driving and driver assistance systems. However, trial-and-error learning and the black-box character of neural networks make it prone to accidental damage in safety-critical environments. We propose to learn a safe vehicle following controller with deep reinforcement learning by imposing state-specific safe sets as output constraints on the policy and call the approach ACC 4S. The main safety goal is the avoidance of rear-end collisions with the front vehicle. To achieve this, we build on the Responsibility-Sensitive Safety model and derive an upper bound for the demanded acceleration. Further limitations emerge from regulatory standards and system limits. We end up with state-specific intervals of safe actions, the safe sets. To impose these safe sets as hard output constraints on the policy, we leverage the recently proposed neural network architecture ConstraintNet. We compare ConstraintNet with an unconstrained neural network, additional clipping as postprocessing, and clipping as part of the neural network. The results show, that the proposed safe sets ensure collision avoidance and ConstraintNet shows superior performance compared to the other approaches.