Distributed Beam Selection for Millimeter-Wave Cell-Free Massive MIMO Based on Multi-Agent Deep Reinforcement Learning

In this paper, we propose a distributed solution to address the beam selection problem in millimeter-wave (mmWave) Cell-Free Massive multiple-input multiple-output (CF-mMIMO) systems. To realize the real-time optimization under dynamic environments with high performance and low complexity, we formulate the sum-rate maximization problem in mmWave CF-mMIMO as a Markov decision process and leverage deep reinforcement learning (DRL) to optimize the beam decision. Furthermore, we propose a distributed solution via multi-agent DRL (MA-DRL) framework to handle the extremely high action dimension and reduce the fronthaul requirements for communications between the access points (APs). Numerical simulations demonstrate the superiority of the proposed solution in the real-time sum-rate performance for mobile users as well as the fronthaul requirements reduction brought by the distributed architecture.