Accelerating Multi-Agent DDPG on CPU-FPGA Heterogeneous Platform

Multi-Agent Reinforcement Learning (MARL) is a key technology in artificial intelligence applications such as robotics, surveillance, energy systems, etc. Multi-Agent Deep Deterministic Policy Gradient (MADDPG) is a state-of-the-art MARL algorithm that has been widely adopted and considered a popular baseline for novel MARL algorithms. However, existing implementations of MADDPG on CPU and CPU-GPU platforms do not exploit fine-grained parallelism between cooperative agents and handle inter-agent communication sequentially, leading to sub-optimal throughput performance in MADDPG training. In this work, we develop the first high-throughput MADDPG accelerator on a CPU-FPGA heterogeneous platform. Specifically, we develop dedicated hardware modules that enable parallel training of each agent's internal Deep Neural Networks (DNNs) and support low-latency inter-agent communication using an on-chip agent interconnection network. Our experimental results show that the speed performance of agent neural network training improves by a factor of
$3.6\times-24.3\times$
and
$1.5\times-29.5\times$
compared with state-of-the-art CPU and CPU-GPU implementations. Our design achieves up to a
$1.99\times$
and
$1.93\times$
improvement in overall system throughput compared with CPU and CPU-GPU implementations, respectively.