Large-Scale Multi-Agent System Optimization with Fixed Final Density Constraints: An Imbalanced Mean-Field Game Theory

This paper presents a novel distributed optimization algorithm for large-scale multi-agent systems (LS-MAS), particularly with a given fixed final density constraint. Although the Mean field game (MFG) theory provides a distribution solution to overcome the “Curse of dimensionality” in LS-MAS, it significantly sacrifices LS-MAS optimality and also not be capable of achieving arbitrary fixed final probability density function (PDF) constraint. To overcome these challenges, a novel Imbalanced Mean-Field Game (Imb-MFG) theory is developed along with an adaptive PDF decomposition algorithm and distributed reinforcement learning. Specifically, an induction-based PDF parameter estimation is developed to decompose the final density constraints into multiple imbalanced norm distributions. Then, the Imb-MFG theory is designed by integrating multi-group MFG with a constrained K-means clustering algorithm. To solve the developed Imb-MFG and further obtain the distributed optimal solution, a multi-actor-critic-mass (Multi-ACM) algorithm is designed to learn the solution of multi-group coupled Hamilton-Jacobi-Bellman (HJB) and Fokker-Planck-Kolmogorov (FPK) equations simultaneously. Finally, the convergence of the developed Multi-ACM algorithm is guaranteed through Lyapunov analysis.