Hierarchical Joint Control for Urban Mixed-Autonomy Traffic Optimization

With the fast development of autonomous vehicle technologies, the vehicle fleet will be made up of a mixture of human-driven vehicles and autonomous vehicles (AVs) in the coming 20-30 years. To efficiently utilize abundant data to deal with the mixed-autonomy traffic control problem, this paper formulates and approaches the problem using a deep reinforcement learning framework (DRL). DRL is a promising data-driven approach for traffic signal control and AVs control in a large-scale grid. However, traffic control based DRL is quite challenging since the complexity of control and the large search space of the policy. To deal with these issues, we propose a hierarchical joint control framework based on prior knowledge. Specifically, traffic signals at intersections and AVs are controlled by their local controllers, set according to well-adjusted policies; while the coordination of the traffic signals at intersections and the coordination among AVs are determined by two master controllers, respectively. Thus, the control of the whole grid is handled by two master controllers. In this way, the dimension of the action space is greatly decreased and the control operates much smoother. We verify our method by implementing a series of experiments in SUMO. The numerical experiments demonstrate the potential of the mixed-autonomy traffic control, compared with traditional traffic signal systems without AVs. We also demonstrate that our method is easy to train and operates robustly.