Robust Distributed MISO Beamforming Using Multi-Agent Deep Reinforcement Learning

Sum-rate maximizing beamforming with base station (BS) power constraints is an important task for interference management in multicell multiple-input-single-output (MISO) down-link systems. This work considers a distributed implementation for multicell MISO beamforming in the presence of imperfect channel state information (CSI), based on multi-agent deep reinforcement learning. In particular, we develop a multi-agent deep deterministic policy gradient (MADDPG) based algorithm. Specifically, each BS as an agent is provided with an independent DDPG network with convolutional neural network structure, which can learn to choose the beam direction and allocate power for multiple users via a limited information exchange protocol. To achieve robustness against CSI errors, we design a robust reward function based on the averaged data rate and the interference penalty generated by each BS. Simulation results demonstrate the effectiveness of the proposed MADDPG algorithm under large CSI errors.