Digital Twins-based Multi-agent Deep Reinforcement Learning for UAV-assisted Vehicle Edge Computing

UAV-assisted vehicle-edge-computing (VEC) has become a viable solution for a new generation of intelligent transportation systems (ITS) and has attracted widespread attention from academia and industry. Compared with fixed ground devices, UAV can provide line-of-sight (LoS) link and has good mobility, which better matches the needs of individual wireless connectivity and high mobility of vehicle. However, the mobility of UAVs leads to dynamic changes in the network topology environment and brings new challenges in the rational path planning of UAVs, which brings new problems for network autonomous decision-making to achieve network resource allocation and load balancing. Therefore, in order to solve above problems, we introduce digital twins-based multi-agent deep Q-network (DT-based MADQN). Digital twin (DT) collects network data and reconstructs the network environment and provides the basis for Deep reinforcement learning (DRL) model training. DRL model provides a network decision-making solution based on real-time network status and empirical data. The simulation results show the effectiveness of the proposed algorithm. Compared to the baseline algorithm, it reduces the average task delay by 16.4% and improves the task completion rate by 97.6%.

I. Introduction

Unmanned aerial vehicles (UAVs), also known as drones, have attracted a wide range of attention over the past decade in a variety of applications, such as surveillance, aerial imaging, cargo delivery, etc. [1]. They are now widely used in civilian areas such as mobile communications, auxiliary communications, and traffic flow control [2]. Due to flexible mobility and communication coverage capabilities, UAV can provide ubiquitous assisted computing power. In modern cities, there are many types of computing needs for growing smart cars [3]. However, limited base station coverage range and long-distance data transmission can lead to unacceptable transmission delays, which result in the inability to meet the computing requirements with timeliness and quality of service (QoS) requirements. UAV assisted vehicle edge computing (VEC) is a promising solution. By extending the underlying services to the server nodes at the edge, the vehicle can break through the limit of its own computing power and reduce limit of delay tolerance [4]. In addition, UAV can always maintain the line-of-sight (LoS) transmission with the moving vehicle through its own movement, providing a stable and reliable data transmission connection.