Simulation of Vehicle Interaction Behavior in Merging Scenarios: A Deep Maximum Entropy-Inverse Reinforcement Learning Method Combined With Game Theory

Simulation testing based on virtual scenarios can improve the efficiency of safety testing for high-level autonomous vehicles (AVs). In most traffic scenarios, such as merging scenarios, the interactions between vehicles are a game process. Therefore, a critical factor is to accurately simulate the game and interaction processes between the background vehicle (BV) and AV in the test environment. With the increasing availability of natural driving data, a data-driven approach can be introduced to identify the underlying driving behavior patterns in actual driving data. Thus, this article proposes a data-driven method for modeling BV behavior for AV testing in virtual scenarios. The method describes the vehicle decision process in the merging scenario as a standard Markov decision process (MDP). Based on game theory, we considered the BV as a game subject to illustrate the vehicle interaction process. Furthermore, a deep maximum entropy-inverse reinforcement learning combined with the game matrix is proposed to identify the reward function that describes BV behavior. The obtained reward function is used to design a deep Q-network algorithm to simulate the behavior of BV. Finally, the effectiveness and feasibility of the proposed method are verified by comparing it with natural driving data. Moreover, we performed comparative tests with the other two baseline methods; the results show that the proposed method can accurately simulate the interaction behaviors between vehicles in the virtual scenarios.

I. Introduction

Safety verification is critical for autonomous vehicle (AV) development and deployment. The mainstream approach for AV testing involves simulating background vehicles (BVs) that dynamically interact with the tested AVs in various virtual scenarios [1]. To fit the natural driving environment, generating BVs with realistic and interactive driving behavior like a human driver is a hot topic for simulation testers [2], [3].