Model-Free Optimal Control using SPSA with Complex Variables

This work proposes a stochastic optimization algorithm to construct model-free controllers for nonlinear stochastic systems, where both the state and measurement equations are unknown. With no stringent model assumptions imposed on the underlying systems, the controllers are constructed by function approximators (FA), such as neural networks or polynomials. The optimal control refers to the optimal parameters within the FA. Since the functional form of the system remains unknown, we need to estimate the gradient to proceed with gradient-type algorithms. Inspired by the simultaneous perturbation stochastic approximation (SPSA) algorithm and the complex-step gradient approximation, this work proposes a novel estimation for the gradient using only the system measurements. In contrast to prior work that requires at least two system measurements per iteration, the newly proposed method requires only one system measurement per iteration, which makes it suitable in tracking transient systems. We also establish an almost sure convergence result for stochastic approximation algorithms with time-varying objective functions. In the numerical illustration, we consider a non-additive noise model with polynomial controller FA, where the proposed algorithm shows a promising result of converging to the global minimum within a small number of iterations.