Zero-Sum Game (ZSG) based Integral Reinforcement Learning for Trajectory Tracking Control of Autonomous Smart Car

The ultimate aim of our research study is the development, practical implementation, and benchmarking of continuous-time, online reinforcement learning (RL) schemes for the trajectory tracking control (TTC) of fully autonomous vehicles (AVs) in real-world scenarios. The adaptive optimality and model-free nature offered by RL has a stronger promise against its model-based counterparts, such as MPC, against uncertainties related to the vehicle, road, tire-terrain and environmental dynamics. The existing studies on RL based AV control are mostly theoretical, often dealing with high-level TTC, and perform evaluations in simulations considering simplified or linear models with no disturbance and slip effects. The literature also demonstrates the lack of practical implementations in overall RL based autonomous vehicle control. Our ultimate goal is to fill these theoretical and practical gaps by designing and practically evaluating novel RL strategies that will improve the performance of TTC against uncertainties at all levels. This paper presents the simulation results of our preliminary studies in the online, longitudinal tracking control of a realistic AV (with uncertain nonlinear dynamics, as well as disturbance, and slip effects), which we treat as a Zero-Sum Game (ZSG) problem using an Integral Reinforcement Learning (IRL) approach with synchronous actor and critic updates (SyncIRL). The results are promising and motivate the practical implementation of the approach for combined longitudinal and lateral control of AV.

I. Introduction

Trajectory tracking control (TTC) is the motion control layer in AV operation, and plays a critical role in the safety, performance and efficiency of autonomous vehicles (AVs). TTC involves two levels of control; high-level, i.e. calculation of the required force for the desired motion of the vehicle, and low level control to calculate the wheel motor torque that will generate the required force. The high-level control faces uncertainties of the vehicle dynamics and external disturbances, while the low-level control must handle the slip effects, which is a major challenge in vehicle control.