Global Linear Convergence of Online Reinforcement Learning for Partially Observable Systems

In this paper, we propose a Policy Gradient method (PGM) to design an optimal controller for discrete-time partially observable systems. The proposed method utilizes the fact that the state of the system can be statically estimated as an algebraic map of finite-length input-output history data. We show that the PGM including the estimation of this map is globally linear convergent for discrete-time partially observable systems. The proof consists of two-steps: First, we show it for SISO systems by using the fact that the covariance matrix of input-output history for SISO systems is invertible. MIMO systems, however, do not have this property because the state estimation from the history is redundant, in other words, the aforementioned map has to be a horizontal matrix. To overcome this problem, we next introduce the modal decomposition to the convergence analysis, thereby guaranteeing the global linear convergence for generic MIMO systems. The effectiveness of the proposed method is shown through a numerical simulation.