Action Masking-Based Proximal Policy Optimization With the Dual-Ring Phase Structure for Adaptive Traffic Signal Control

Driven by advances in artificial intelligence, deep reinforcement learning (DRL) has made remarkable strides in adaptive traffic signal control (ATSC), empowering improved handling of fluctuating traffic volumes and congestion. However, in most existing studies, trained agents exhibit poor transferability in scenarios with varying vehicle turning ratios, and the switching rules for the signal stage sequence do not align with the actual traffic demands. To address these issues, this paper presents action masking based proximal policy optimization with the dual-ring phase structure (AMPPO-DR), a novel ATSC model based on DRL that can simultaneously optimize the stage sequence and duration. Specifically, we consider the correlation between states and actions and utilize intersection channelization to predict vehicle turning directions. Moreover, we define the action as selecting the next green stage and establish variable-stage-sequence constraint rules on the basis of the dual-ring phase structure. To satisfy the constraints on the stage sequence, we propose the AMPPO algorithm, which dynamically adjusts the policy network outputs to mask invalid stages in real time. The simulation experiments demonstrate that the proposed method can effectively adapt to changing turning flows, enabling flexible and rational stage switching and ultimately increasing traffic efficiency.

I. Introduction

With the emergence of concepts such as the smart city, the intelligent transportation system, and the cooperative vehicle infrastructure system, traditional traffic signal control (TSC) methods are insufficient. Intersection facilities such as cameras, sensors, and induction loops can easily gather road condition data, and precise vehicle navigation systems at the lane level greatly increase the accuracy of vehicle position data. The digitization and informatization of urban traffic systems have greatly facilitated adaptive traffic signal control (ATSC).