Investigating the Effectiveness of Reinforcement Learning in Closed-Loop Systems with Time Delays

Data-driven controllers have gained prominence in diverse control applications, attributed to their inherent flexibility and adaptability to complex system dynamics. However, managing time delays in closed-loop systems remains a significant challenge in their deployment. These delays can arise from various sources, such as computational latency, actuator reaction time, and communication delays. Unaddressed, these time lags can induce system instability and degrade performance. This paper rigorously analyzes the impact of time delays on data-driven controllers and introduces methodologies to mitigate their adverse effects. Specifically, we explore the integration of the Smith predictor with Deep Reinforcement Learning (SP-DRL) to formulate a control law capable of effectively managing both time delays and system uncertainties, while ensuring robust performance. We demonstrate that this DRL-based framework, initially trained in stable environments, generalizes well to unstable systems. Our investigation delineates the scenarios conducive to the successful application of this approach and identifies factors influencing its effectiveness. To substantiate our findings, we present a case study involving a first-order delayed linear system with nonlinear actuation modules. Numerical simulations are employed to compare the robustness of SP-DRL scheme against the DRL standalone and the classical controls, such as PID and Linear Quadratic Regulator (LQR), in the presence of delays.