Centralized Cooperation for Connected Autonomous Vehicles at Intersections by Safe Deep Reinforcement Learning

Connected and automated vehicles (CAVs) have the potential to transform traffic management, especially at intersections. Traditional traffic signals might become obsolete with the implementation of autonomous intersection management (AIM) systems, which aim for efficient and safe vehicle flow. Current AIM methods often rely on optimization control algorithms, which are not computationally efficient. Some methods use reinforcement learning (RL) but compromise safety for rewards and simplify traffic scenarios by designating specific turn lanes. This paper introduces a novel approach, the risk situation-aware constrained policy optimization (RSCPO), to enhance RL training with safety assurance. It uses Kullback-Leibler (KL) divergence to form a trust region, identifying risk levels in policy updates that could lead to dangerous situations, and suggests safe policy update mechanisms. Furthermore, the paper presents a safety reinforced all-directional autonomous intersection management (SafeR-ADAIM) algorithm. This algorithm accounts for the complexity of unpredictable all-direction turn lanes and collaboratively ensures the safety, efficiency, and smooth operation of CAVs at intersections. In simulations, our method surpasses the model predictive control (MPC)-based method in computational and traffic efficiency by 67.81 and 1.46 times, respectively. Additionally, it significantly reduces the mean collision rate from at most 35.01% to 0% compared to non-safety aware RL methods.

I. Introduction

The advancement of autonomous driving and internet of vehicles technology provides a promising opportunity to enhance traffic safety and mobility in intersection management [1]. By leveraging vehicle-to-infrastructure (V2I) communication, a centralized autonomous intersection management (AIM) controller can be installed at the intersection to coordinate the movement of CAVs, guaranteeing their efficient and conflict-free passage, and optimizing the ride comfort. This approach can improve upon the unnecessary and unfair waiting times caused by the coarse-grained management of traffic signals. AIM has garnered significant research interest in recent years. Traditionally, these AIMs handle potential conflicts based on control strategies such as rule-based, optimization-based or machine learning-based methods to prevent anticipated conflicts from occurring [6-39].