Centralized Training with Decentralized Execution Reinforcement Learning for Cooperative Multi-agent Systems with Communication Delay

In cooperative multi-agent systems, efficient coordination among agents is important when accomplishing tasks. VFFAC is a method that learns the communication system between agents and their interactions with the environment to obtain policies with high performance. However, this method results in decreased performance of policy in environments with a delay in communication. Furthermore, there is no formulation of the control problem of a cooperative multi-agent system with communication delays in unknown environments. In this study, we formulated a decision-making problem in a cooperative multi-agent system with an unknown environment model and a certain length delay in communication. We also propose a method to handle communication delays by using the history of information obtained through communication. We demonstrated that the proposed method successfully learns policy with high rewards through simulated experiments in an environment with a communication delay.

1. INTRODUCTION

Reinforcement learning (RL) [1] is a framework of machine learning in which an agent learns effective policies to accomplish a task by trial and error in an unknown environment. The agent obtains rewards through trial and error against the environment, and aims to identify a policy that maximizes the discounted cumulative reward.