Mean Field Deep Reinforcement Learning for Fair and Efficient UAV Control

Unmanned aerial vehicles (UAVs) can provide flexible network coverage services. UAVs can be applied in a large number of scenarios, such as emergency communication and network access in areas without terrestrial network coverage. However, UAVs are limited to relatively short communication range and restricted energy resources. In extreme conditions such as disasters, there may also be a problem that the communication bandwidth is limited and the UAV cannot communicate with the server with a large amount of information, so a decentralized solution is expected. In addition, the interaction between multiple objectives and multiple UAVs leads to a huge state space, which makes large-scale practical applications difficult. To simplify complex interactions, we modeled the UAV control problem with mean-field game (MFG). We propose a new UAV control method, the mean-field trust region policy optimization (MFTRPO), which uses the MFG method to construct the Hamilton–Jacobi–Bellman/Fokker–Planck–Kolmogorov equation that obtains the optimal solution and solves the difficulties in the practical application through the trust region policy optimization and neural network feature embedding methods. The proposed method: 1) maximizes communication efficiency while ensuring fair communication range and network connectivity; 2) fuses the mean-field theory with deep reinforcement learning techniques; and 3) is scalable and adaptive. We conduct extensive simulations for performance evaluation. The simulation results have shown that MFTRPO significantly and consistently outperforms two commonly used baseline methods in terms of coverage, fairness, and energy consumption.

I. Introduction

The emerging new networks, e.g., the Internet of Things (IoT) and the unmanned aerial vehicle (UAV) networks, become more decentralized and autonomous. UAVs can serve as aerial base stations (UAV-BSs) to expand coverage and enhance the performance of communication networks [1], such as emergency communication during natural disasters and network access in areas without good terrestrial network coverage. When the communication network is damaged by natural disasters, the UAVs provide fast and reliable communication guarantee for ground users [2], which is illustrated in Fig. 1. For example, in August 2017, after the Jiuzhaigou earthquake in China, UAV-BSs were applied and used for emergency communication in the Heze Village of the Jiuzhaigou scenic area where the cable was damaged and the terrestrial BSs were interrupted. UAV-BSs quickly provided the mobile communication signals to the affected area of more than 30 km². Fig. 1.

UAV network providing communication coverage for ground users in a target region.