Distributed Q-Learning for Dynamically Decoupled Systems

Control of large-scale networked systems often necessitates the availability of complex models for the interactions amongst the agents. However, in many applications building accurate models of these interactions might be prohibitive due to the curse of dimensionality or their inherent complexity. In the meantime, data-guided control methods can circumvent model complexity by directly synthesizing the controller from the observed data. In this paper, we propose a distributed Q-learning algorithm to design a feedback mechanism given an underlying graph structure parameterizing the agents' communication. We assume that the distributed nature of the system arises from a common cost and show that for the particular case of identical dynamically decoupled systems, the learned controller converges to the optimal Linear Quadratic Regulator controller for each subsystem. We provide a convergence analysis and verify the result with an example.