Adaptive optimal controllers based on Generalized Policy Iteration in a continuous-time framework

In this paper we present two adaptive algorithms which offer solution to the continuous-time optimal control problem for nonlinear, affine in the inputs, time-invariant systems. Both algorithms were developed based on the generalized policy iteration technique and involve adaptation of two neural network structures namely actor, providing the control signal, and critic, performing evaluation of the control performance. Despite the similarities, the two adaptive algorithms differ in the manner in which the adaptation takes place, required knowledge on the system dynamics, and formulation of the persistence of excitation requirement. The main difference is that one algorithm uses sequential adaptation of the actor and critic structures, i.e. while one is trained the other one is kept constant, while for the second algorithm the two neural networks are trained synchronously in a continuous-time fashion. The two algorithms are described in detail and proof of convergence is provided. Simulation results of applying the two algorithms for finding the optimal state feedback controller of a nonlinear system are also presented.

I. Introduction

From a mathematical point of view the solution of the optimal control problem is connected with the solution of the underlying Hamilton-Jacobi-Bellman (HJB) equation. Numerous results are available for obtaining the solution of the discrete-time HJB equation in a forward in time fashion (see for example [4], [2] and the references therein). In the continuous-time (CT) case, of interest in this paper, things become more complicated due to the nature of this nonlinear differential equation. Considerable advances, in the form of algorithms which approximately solve the CT HJB equation for general nonlinear systems, have only recently been made (e.g. [1], [3], [10]).