Double Deep Q-Learning and SAC Based Hybrid Beamforming for 5G and Beyond Millimeter-Wave Systems

Fully digital beamforming techniques are costly and power hungry when employed with massive multiple input multiple output (MIMO) systems at the millimeter-Wave (mmWave) bands. Hybrid beamforming, which uses a very few number of radio frequency chain with a network of phase shifters, is a cost- and an energy-efficient alternative beamforming solution. However, the unit modulus constraint imposed by the phase shifters makes this problem inherently nonconvex, therefore may induce unaffordable computational complexity. Hence, designing hybrid beamforming algorithms which further improve the spectral efficiency, hardware efficiency, and computational efficiency is of crucial importance. Therefore, in this paper, we aim at solving hybrid beamforming using the theory of deep reinforcement learning, which has been successful in solving several high-dimensional nonconvex problems. Although deep reinforcement learning has been previously proposed, nearly all prior studies focus on Q-learning, which is known to suffer from overestimation bias. Different from these previous studies, we propose to solve hybrid beamforming for a single-user massive MIMO (SU-MIMO) using two methods 1) a double deep Q- learning with replay experience and soft target network updates method for discrete action space setting; 2) a soft actor critic method for continuous action space setting. These models are tested using numerical simulations and evaluated using the spectral efficiency and computational efficiency. This paper shows that these models can achieve a near-optimal performance in terms of the spectral efficiency while reducing the computational complexity.