HRL-based Joint Band Assignment and Beam Management in 4G/5G Multi-Band Networks

Incorporating the sub-6 GHz band into 5G networks can improve data rates by leveraging the benefits of propagation across frequency ranges. Sequential decision making algorithms such as deep reinforcement learning (DRL) can adaptively select a band over time to take full advantage of the multi-band operation. The distinctive beam management procedure between the sub-6 GHz band and the millimeter wave (mmWave) band, though, pose a sample efficiency challenge for DRL algorithms. In this paper, we use hierarchical reinforcement learning (HRL) to divide and conquer the joint band assignment and beam management problem. The proposed HRL-based method uses rate feedback for intermittent band determination and frequent beam management mode decisions. We show with numerical evaluation that the proposed algorithm achieves a quicker increase in data rate compared to baselines and identify off-policy correction methods as a key factor for this enhancement.