Graded-Q Reinforcement Learning with Information-Enhanced State Encoder for Hierarchical Collaborative Multi-Vehicle Pursuit

The multi-vehicle pursuit (MVP), as a problem abstracted from various real-world scenarios, is becoming a hot research topic in the Intelligent Transportation System (ITS). The combination of Artificial Intelligence (AI) and connected vehicles has greatly promoted the research development of MVP. However, existing works on MVP pay little attention to the importance of information exchange and cooperation among pursuing vehicles under the complex urban traffic environment. This paper proposed a graded-Q reinforcement learning with information-enhanced state encoder (GQRL-IESE) framework to address this hierarchical collaborative multi-vehicle pursuit (HCMVP) problem. In the GQRL-IESE, a cooperative graded Q scheme is proposed to facilitate the decision-making of pursuing vehicles to improve pursuing efficiency. Each pursuing vehicle further uses a deep Q network (DQN) to make decisions based on its encoded state. A coordinated Q optimizing network adjusts the individual decisions based on the current environment traffic information to obtain the global optimal action set. In addition, an information-enhanced state encoder is designed to extract critical information from multiple perspectives and uses the attention mechanism to assist each pursuing vehicle in effectively determining the target. Extensive experimental results based on SUMO indicate that the total timestep of the proposed GQRL-IESE is less than other methods on average by 47.64%, which demonstrates the excellent pursuing efficiency of the GQRL-IESE. Codes are outsourced in https://github.com/ANT-ITS/GQRL-IESE.

I. Introduction

The Intelligent Transportation System (ITS), as an essential part of the smart city, is greatly facilitated by the development of emerging technologies. The Internet of Vehicles (IoVs) enables ITS to realize dynamic and intelligent management of traffic [1] [2]. Pursuit-evasion game (PEG), as a realistic problem for studying the self-learning and autonomous control of multiple agents, has been extensively studied in many fields, such as spacecraft control [3] and robot control [4]. Multi-vehicle pursuit (MVP), as an embodiment of PEG in ITS, has more conditional constraints, such as complex road structures, additional traffic participants, and traffic rules constraints. A patrol guide released by the New York City Police Department representatively describes an MVP game, where multiple policy vehicles cooperate to capture one or multiple suspected vehicles [5].