Distributed Multi-Agent Reinforcement Learning for Cooperative Low-Carbon Control of Traffic Network Flow Using Cloud-Based Parallel Optimization

The escalating air pollution resulting from traffic congestion has necessitated a shift in traffic control strategies towards green and low-carbon objectives. In this study, a graph convolutional network and self-attention value decomposition-based multi-agent actor-critic (GSAVD-MAC) approach is proposed to cooperative control traffic network flow, where vehicle carbon emission and traffic efficiency are considered as reward functions to minimize carbon emissions and traffic congestions. In this method, we design a local coordination mechanism based on graph convolutional network to guide the multi-agent decision-making process by extracting spatial topology and traffic flow characteristics between adjacent intersections. This enables distributed agents to make low-carbon decisions which not only account for their own interactions with the environment but also consider local cooperation with neighboring agents. Further, we design a global coordination mechanism based on self-attention value decomposition to guide multi-agent learning process by assigning various weights to distributed agents with respect to their contribution degrees. This enables distributed agents to learn a globally optimal low-carbon control strategy in a cooperative and adaptive manner. In addition, we design a cloud computing-based parallel optimization algorithm for the GSAVD-MAC model to reduce calculation time costs. Simulation experiments based on real road networks have verified the advantages of the proposed method in terms of computational efficiency and control performance.

I. Introduction

Traffic congestions and exhaust emissions have become one of the sources restricting the development of smart cities. Numerous adaptive traffic signal control (ATSC) algorithms have been proposed to optimize traffic efficiency and traffic safety by adjusting signal timing scheme [1], [2]. Multi-agent deep reinforcement learning (MADRL) [3], [4] is a promising method to solve the complicated multi-intersection ATSC problem, which incorporates the decision-making capacity of reinforcement learning, the perception capacity of deep learning and the coordination capacity of multi-agent. Even so, most of the existing MADRL for ATSC focus on traffic efficiency improvement while neglecting the serious effect of carbon emissions to the environment [5], [6], [7]. The driving behaviors of abrupt acceleration and deceleration and the irrational timing strategies of traffic signals are the main causes of increasing energy consumption and exhaust emissions. Some experts and scholars have focused on optimizing ecological driving behaviors, which attempts to reduce fuel consumption and emissions by smoothing driving speed curves of vehicles approaching intersections [8], [9]. However, they overlook the impacts of signal timing approaches on vehicle energy consumption and carbon emissions. Different from previous studies, MADRL will be utilized in this study to minimize carbon emissions and traffic congestions through cooperative control of timing strategies at distributed multiple signalized intersections.