Distributed Reinforcement Learning For Swarm Systems With Reward Machines

We introduce a decentralized reinforcement learning (RL) algorithm for swarm systems where reward machines (a type of Mealy machine) are used to encode the non-Markovian reward functions. We use the generalized moments (GMs) to characterize swarm features. Each agent estimates the GM of the swarm state and uses it to estimate the state of the reward machine. The agent then uses the estimated state of the reward machine to update its q-values. We use the gossip algorithm for communication between the agents. Agents exploit this communication to update their estimated GM of the swarm state, which leads to an update of their estimated state of the reward machine. We derive an upper bound for the error between the estimate GM of the swarm state and the ground truth GM of the swarm state, and using that upper bound; we prove the convergence of our proposed algorithm, swarm q-Iearning with reward machines (Swarm-QRM). To demonstrate the efficiency of our approach, we present two case studies wherein, for Case Study 1, a swarm of agents will perform a pickup and delivery task, and in Case Study 2, the swarm of agents will conduct a search and rescue task. We also show that Swarm-QRM outperforms the baselines, q-Iearning in augmented state space (QAS), and Double Deep Q-Network (DDQN).