Joint Beam Training and Data Transmission Control for mmWave Delay-Sensitive Communications: A Parallel Reinforcement Learning Approach

Future communication networks call for new solutions to support their capacity and delay demands by leveraging potentials of the millimeter wave (mmWave) frequency band. However, the beam training procedure in mmWave systems incurs significant overhead as well as huge energy consumption. As such, deriving an adaptive control policy is beneficial to both delay-sensitive and energy-efficient data transmission over mmWave networks. To this end, we investigate the problem of joint beam training and data transmission control for mmWave delay-sensitive communications in this paper. Specifically, the considered problem is firstly formulated as a constrained Markov Decision Process (MDP), which aims to minimize the cumulative energy consumption over the whole considered period of time under delay constraint. By introducing a Lagrange multiplier, we transform the constrained MDP into an unconstrained one, which is then solved via a parallel-rollout-based reinforcement learning method in a data-driven manner. Our numerical results demonstrate that the optimized policy via parallel-rollout significantly outperforms other baseline policies in both energy consumption and delay performance.

I. Introduction

With the fast development of mobile Internet and electronic devices, various emerging applications such as virtual reality and augmented reality have entered into our society and have created an explosive growth in the data volume. On the other hand, due to the heavily crowded and fragmented spectrum below 6 GHz, it becomes more and more challenging to meet this ever-increasing data rate demand using current wireless networks, which are mainly operated at sub-6 GHz frequency band. By contrast, millimeter wave (mmWave) band, ranging from 30 GHz to 300 GHz, is widely considered as a key technology to achieve multi-gigabit data transmission thanks to its large available bandwidth [1], [2]. But unfortunately, mmWave communications often suffer severe propagation loss. To address this concern, mmWave transceivers are usually equipped with large antenna arrays for beamforming, which can compensate for the severe propagation loss and guarantee a favorable signal-to-noise ratio (SNR) for mmWave communications [3], [4].