SafeDriveRL: Combining Non-Cooperative Game Theory with Reinforcement Learning to Explore and Mitigate Human-Based Uncertainty for Autonomous Vehicles

Increasingly, artificial intelligence (AI) is being used to support automotive systems, including autonomous vehicles (AVs) with self-driving capabilities. The premise is that learning-enabled systems (LESs), those systems that have one or more AI components, use statistical models to make better informed adaptation decisions and mitigate potentially dangerous situations. These AI techniques largely focus on uncertainty factors that can be explicitly identified and defined (e.g., environmental conditions). However, the unexpected behavior of human actors is a source of uncertainty that is challenging to explicitly model and define. In order to train a learning-enabled AV, developers may use a combination of realworld monitored data and simulated external actor behaviors (e.g., human-driven vehicles, pedestrians, etc.), where participants follow defined sets of rules such as traffic laws. However, if uncertain human behaviors are not sufficiently captured during training, then the AV may not be able to safely handle unexpected behavior induced by human-operated vehicles (e.g., unexpected sudden lane changes). This work introduces a non-cooperative game theory and reinforcement learning-based (RL) framework to discover and assess an AV's ability to handle high-level uncertain behavior(s) induced by human-based rewards. The discovered synthetic data can then be used to reconfigure the AV to robustify onboard behaviors.

1 Introduction

Recently, significant and rapid advancements in deep learning (e.g., object detection algorithms for onboard cameras) have facilitated their use in autonomous vehicles (AVs) [1]. These learning-enabled systems (LESs) are safety critical [19], where their failure can result in loss of life, injuries, and financial damage.¹

This paper uses the term LESs to refer to AI-based systems whose behaviors are optimized based on training experiences (e.a., object detection algorithms).

LESs may not operate as expected if the training data does not sufficiently capture run-time contexts [22]. For example, an AV may be trained with external agents such as pedestrians or other vehicles, whose training behavior conforms to defined rules [39]. In contrast, human agents may exhibit unexpected behaviors (e.g., suddenly braking, unexpected lane changes, etc.), thereby introducing a new source of uncertainty for AVs. This paper uses the term “ego vehicle” to describe the AV under study and “non-ego vehicle” to describe other vehicles. We propose a non-cooperative game theory²

Non-cooperative game theory describes a game theory setup where players are only aware of and attempt to optimize their individual objectives [30].

[30] and reinforcement learning (RL) [37] framework to discover unexpected behavior exhibited by a trained (selfish) non-ego vehicle and the unexpected responses from a naïve³

Basic driving behavior with no experience with uncertainty.

ego vehicle, which can be used to enable the robustification of the ego vehicle to prevent and/or mitigate unsafe situations.