Multi-Agent Q-Learning and Max-Pressure Control for Large-Scale Traffic Signal Control

Traffic congestion is becoming an increasingly critical issue, requiring urgent and effective solutions. As a data-driven method, reinforcement learning (RL) enables large-scale adaptive traffic signal control (TSC) in transportation systems. The decentralized multi-agent reinforcement learning (MARL) framework proves effective in scenarios involving multiple intersections. However, there is a lack of an effective behavior policy for agents to explore actions in the TSC scenario. A novel behavior policy is proposed by introducing the max-pressure control method in the transportation field into the $\epsilon$-greedy behavior policy. An agent selects an action with the maximum pressure phase to maximize intersection traffic throughput, enabling each agent to choose an appropriate action according to real-time intersection traffic conditions in the TSC scenario. A multi-agent Q-learning algorithm utilizes the novel behavior policy for large-scale TSC. Experimental results show that compared to other existing decentralized MARL algorithms, the proposed approach performs well on a large synthetic grid under dynamic simulated traffic peak conditions.

I. Introduction

The number of vehicles has increased rapidly in recent years, which bring about a series of problems such as traffic congestion and environmental pollution in urban areas. Eliminating traffic congestion has become a hot issue to be solved urgently. Traffic signal control (TSC) [1] is one of the most common methods for effectively regulating traffic order, ensuring traffic safety, and improving traffic efficiency. According to the scope of controlled intersections, it can be divided into three different control types: spot control [2], line control [3], and plane control [4]. Spot control is signal control for a single intersection to ensure orderly traffic at the intersection. Line control is coordinated control of multiple consecutive intersections on a road to improve the traffic efficiency of the road. Plane control is the overall control of intersections to achieve the optimal operation of regional traffic, which is a challenging issue because urban traffic has high complexity.