Learning-based Delay Optimization for Self-Backhauled Millimeter Wave Cellular Networks

Multihop self-backhauling is considered to be a key enabling technology for millimeter wave cellular deployments. We consider the multihop link scheduling problem with the objective of minimizing the end-to-end delay experienced by a typical packet. This is a complex problem, and so we model the system as a network of queues and formulate it as a Markov decision process over a continuous action space. This allows us to leverage the deep deterministic policy gradient algorithm from reinforcement learning to learn a delay minimizing scheduling policy under two scenarios: (i) an ideal setup where a centralized scheduler performs all per slot scheduling decisions and has full instantaneous knowledge of network state and (ii) again a centralized scheduler, but network state feedback and scheduling decisions are limited to once per frame, i.e. per many slots. For the second scenario, we model the scheduler by a recurrent neural network, which allows the agent to understand the evolution of the network state over the frame. Detailed system- level simulations show that the proposed scheduler outperforms the backpressure scheduler and the max-min delay round-robin scheduler in terms of packet delay, particularly for the more realistic second scenario.