Multi-Agent Deep Reinforcement Learning for Massive Access in 5G and Beyond Ultra-Dense NOMA System

With the rapid development of machine-type communications (MTC), the future communication architecture needs to provide services for both human-type communications (HTC) and MTC with unique characteristics. The huge connections from MTC bring serious challenges to the existing wireless network. Ultra-dense network (UDN), a promising candidate technology, can support massive device access through dense deployment of small base stations (SBSs). Different from the resource management in traditional wireless network with single base station (BS), the resource allocation problem at BS level is more prominent in UDN, and the diversity of devices will make this problem more complicated. In view of this, we investigate the joint optimization of massive access and resource management in the UDN where HTC and MTC coexist. Considering the computational complexity and scalability, we propose a multi-agent deep reinforcement learning based SBS state selection scheme, in which each SBS acts as an agent and selects the optimal state between active and idle by continuously interacting with the environment. In addition, we adopt the power-domain non-orthogonal multiple access to further improve system throughput, and use grant-based and grant-free access manners for HTC and MTC respectively, so as to meet their unique characteristics. Extensive numerical results demonstrate the superior performances of proposed scheme in multiple perspectives.

I. Introduction

In recent years, with the rapid development of Internet of Things (IoT) applications (such as smart home, smart grid, smart factory, etc.) and wireless cellular communication technology (e.g., 5G and beyond), the number of IoT devices has increased exponentially. According to the Cisco Annual Internet Report [1], from 2018 to 2023, the total number of IoT devices will increase from 18.4 billion to 29.3 billion, and the proportion of machine-type communication devices (MTCDs) (e.g., environmental sensors, smart meters, wearable devices, etc.) will increase from 33% to 50%. It is obvious that the number of MTCDs will soon exceed the number of human-type communication devices (HTCDs) (such as smart phones, tablets, etc.) and become the mainstream.