Adaptive Critic Learning and Experience Replay for Decentralized Event-Triggered Control of Nonlinear Interconnected Systems

In this paper, we develop a decentralized event-triggered control (ETC) strategy for a class of nonlinear systems with uncertain interconnections. To begin with, we show that the decentralized ETC policy for the whole system can be represented by a group of optimal ETC laws of auxiliary subsystems. Then, under the framework of adaptive critic learning, we construct the critic networks to solve the event-triggered Hamilton-Jacobi-Bellman equations related to these optimal ETC laws. The weight vectors used in the critic networks are updated by using the gradient descent approach and the experience replay (ER) technique together. With the aid of the ER technique, we can conquer the difficulty arising in the persistence of excitation condition. Meanwhile, by using classic Lyapunov approaches, we prove that the estimated weight vectors used in the critic networks are uniformly ultimately bounded. Moreover, we demonstrate that the obtained decentralized ETC can force the overall system to be asymptotically stable. Finally, we present an interconnected nonlinear plant to validate the proposed decentralized ETC scheme.

I. Introduction

Adaptive critic learning (ACL), also known as adaptive critic design, has been a powerful technique to solve optimization problems [1]–[3]. The success of ACL in solving optimization problems mainly relies on an actor–critic structure. In this structure, the actor performs a control policy to systems or environments, and the critic evaluates the cost caused by that control policy and provides reward/punishment signals to the actor. A significant advantage of the actor–critic structure is that, by employing actor–critic dual neural networks (NNs), it can be utilized to avoid the well-known “curse of dimensionality.” In the computational intelligence community, the actor–critic structure is a typical architecture used in adaptive dynamic programming (ADP) [4] and reinforcement learning (RL) [5]. Because ADP and RL have much in common with ACL (e.g., the same implementation structure), they are generally considered as synonyms for ACL. In this paper, we take ADP and RL as the members of ACL family. Over the past few decades, many kinds of ADP and RL have been introduced to handle optimal control problems, such as goal representative ADP [6], Hamiltonian-driven ADP [7], policy/value iteration ADP [8]–[10], robust ADP [11], [12], online RL [13]–[15], and off-policy RL [16]–[18]. Recently, based on the work of Lin [19] building a relationship between the robust control and the optimal control, ACL was successfully applied to solve the robust control problems [20]–[22]. However, when implementing these ACL algorithms, most of them required the controlled systems to be persistently exciting. Unfortunately, it is often intractable to verify the persistence of excitation (PE) condition, especially for nonlinear systems.