Activation Control of Base Stations Based on Multi-agent DQN for Heterogeneous Networks

In this paper, we investigate a method for activation/deactivation control of base stations (BSs) based on a deep Q-network (DQN) that tracks changes in user distribution in heterogeneous networks to increase downlink system throughput. Conventional BS activation control based on deep reinforcement learning (RL) assumes centralized control and represents the entire system with a single action value function. However, this method cannot be applied to dense heterogeneous networks with many BSs because the total number of actions grows exponentially. In the proposed method, each BS is an agent with its own action value function making the DQN-based BS activation control applicable to dense small-cell networks. Several definitions of states are compared from the viewpoints of the load required for acquiring the state variables and the achievable throughput performance. Computer simulations show that the proposed method achieves higher throughput compared to the conventional probabilistic control method and that it is adaptable to different system conditions.