A Survey on Recent Advancements in Autonomous Driving Using Deep Reinforcement Learning: Applications, Challenges, and Solutions

Autonomous driving (AD) endows vehicles with the capability to drive partly or entirely without human intervention. AD agents generate driving policies based on online perception results, which are crucial to the realization of safe, efficient, and comfortable driving behaviors, particularly in high-dimensional and stochastic traffic scenarios. Currently, deep reinforcement learning (DRL) techniques to derive and validate AD policies have witnessed vast research efforts and have shown rapid development in recent years. However, a comprehensive interpretation and evaluation of their strengths and limitations concerning the full-stack AD tasks remain uncharted. This paper presents a survey of this body of work, which is conducted at three levels. First, it analyzes the multi-level AD task characteristics and delves deeply into the current DRL methodologies primarily employed in AD. Second, a taxonomy of the literature studies is constructed from the system perspective, identifying six modes of DRL model integration into an AD architecture that span the entire spectrum of AD policy processes, from perception understanding and decision-making to motion control, as well as verification and validation. Each literature review comprehensively encompasses the main elements of designing such a system, including modeling partially observable environments, state and action spaces, reward structuring, and the design and training methodologies of neural network models. Finally, an in-depth foresight is conducted on how the eight critical issues of AD application development are addressed by the DRL models tailored for real-world AD challenges.

I. Introduction

Recently, advancements in Artificial Intelligence (AI)-equipped vehicle sensors and electronic control units have significantly enhanced the ability to perform a variety of tasks, ranging from object detection [1] and localization [2] to tracking and decision-making across different automation levels. These developments have fine-tuned the perception and computation aspects of Autonomous Driving (AD), facilitating the widespread production of advanced driver assistance systems like predictive emergency braking [3], lane change assistance [4], adaptive cruise control [5], highway pilot [6] and so on. While AD systems have the potential to mitigate accidents caused by factors such as impaired, speeding, reckless, and distracted driving, which continue to contribute to numerous traffic accidents, it is important to consider the possibility of new types of accidents or the exacerbation of existing ones [7]. Moving towards fully AD could greatly reduce such incidents and enhance traffic efficiency, comfort, and energy savings. To date, significant efforts and initiatives have been launched by major industries to break through the assistance and move towards Society of Automotive Engineers (SAE) Level 4 and above [8]. Yet, the journey towards fully autonomous systems adept at navigating complex real-world conditions continues to be a challenging frontier in technology.