Online Optimal Control of Affine Nonlinear Discrete-Time Systems With Unknown Internal Dynamics by Using Time-Based Policy Update

In this paper, the Hamilton-Jacobi-Bellman equation is solved forward-in-time for the optimal control of a class of general affine nonlinear discrete-time systems without using value and policy iterations. The proposed approach, referred to as adaptive dynamic programming, uses two neural networks (NNs), to solve the infinite horizon optimal regulation control of affine nonlinear discrete-time systems in the presence of unknown internal dynamics and a known control coefficient matrix. One NN approximates the cost function and is referred to as the critic NN, while the second NN generates the control input and is referred to as the action NN. The cost function and policy are updated once at the sampling instant and thus the proposed approach can be referred to as time-based ADP. Novel update laws for tuning the unknown weights of the NNs online are derived. Lyapunov techniques are used to show that all signals are uniformly ultimately bounded and that the approximated control signal approaches the optimal control input with small bounded error over time. In the absence of disturbances, an optimal control is demonstrated. Simulation results are included to show the effectiveness of the approach. The end result is the systematic design of an optimal controller with guaranteed convergence that is suitable for hardware implementation.

I. Introduction

Neural Networks (NNs) have been widely used in the controller designs for nonlinear systems where stability is typically the only consideration [1], [2]. Optimality, which is difficult to achieve, is generally preferred for the control of nonlinear systems than stability alone. The optimal control of linear systems accompanied by quadratic cost functions can be achieved by solving the well-known Riccati equation [3], however, the optimal control of nonlinear discrete-time systems often requires solving the nonlinear Hamilton–Jacobi–Bellman (HJB) equation, which does not have a closed-form solution. To extend the results of linear optimal control theory to nonlinear systems, the state dependent Riccati equation (SDRE) [4] was proposed for suboptimal control under certain tight assumptions including the need for full system dynamics.