Applications in Traffic Signal Control: A Distributed Policy Gradient Decomposition Algorithm

This article explores the application of the multiagent reinforcement learning (MARL) algorithm in addressing the large-scale traffic signal control (TSC) problem. To address the TSC problem in complex urban traffic networks, most existing algorithms focus on optimizing local traffic flow at each intersection through decentralized training based on either the local observations or messages from its neighboring intersections, which, however, lacks the concept of cooperative learning. To conquer such limitations, a novel distributed critic with decentralized actor (DCDA) framework is proposed, which allows the communication messages and temporal difference (TD) losses to be exchanged among neighboring intersections. Specially, by considering the traffic network as a communication network of agents (more precisely, intersections and lanes are considered as agents and edges, respectively), a distributed global average TD loss estimation algorithm is designed in the distributed critic step to estimate the global average TD loss estimation and enhance collaboration among agents. Moreover, in the decentralized actor step, the policy gradient decomposition method is adopted for each agents to learn its local policy solely based on its local action-value function. By adhering to the DCDA framework, a novel distributed policy gradient decomposition (DPGD) algorithm is further proposed to address the TSC problem. Empirical experiments demonstrate that the efficiency, robustness, and stability of the DPGD algorithm outperform the state-of-the-art MARL algorithms in both the environments of cooperative adaptive cruise control and adaptive traffic signal control.