A Model-Free Solution for Stackelberg Games Using Reinforcement Learning and Projection Approaches

The Stackelberg game is adopted in many robotics applications. It features a dynamic multi-player setup based on a leader-follower structure. The main challenge involves implementing model-free strategies that can effectively respond to unstructured environments in a data-driven manner. This paper presents a model-free method for solving the Stackelberg game in real-time, wherein the follower's strategy assumes knowledge of the leader's tactics. Moreover, the strategies are implemented in real-time without knowledge of the players' dynamics. The optimization goals are expressed through coupled Bellman optimality equations, highlighting the dependency between leader and follower strategies. The method utilizes a linear adaptive critics framework, where the actor-critic weights are adjusted using a projection method to ensure stability and convergence. This approach is evaluated on systems with delays and unstructured disturbances to demonstrate its robustness.