A Reinforcement Learning Based Service Scheduling Algorithm for Internet of Drones

Originally invented by the military for warfighting, drones have broken through adamant barriers established by traditional commercial and civilian industry and are quickly becoming an accepted part of mainstream. In order to enable drone technology to reach its full potential and integrate heterogeneous drones into existing workflows, Internet of Drones (IoD) has been proposed as a future aerial-ground communication architecture, where drones frequently contact Zone Service Providers (ZSPs) for up-to-date information. When many drones intend to access data through a ZSP concurrently, service scheduling plays a significant role in improving data accessibility. In practice, however, the limited bandwidth and coverage range of ZSP and the high speed of drones make the problem of service scheduling challenging. In this paper, we propose a reinforcement learning based service scheduling algorithm, also called RELESS, to optimally satisfy the service requests of drones in the IoD. In RELESS, the interaction between the ZSP and drones is formulated as a Markov decision process (MDP) which will be solved by the Q-learning algorithm to produce an optimal service scheduling policy. During this process, the ZSP adopts an ∊-greedy exploration method to continuously fine-tune its service scheduling policy with various system states, which is guaranteed to converge to an optimal policy. We develop a discrete-event driven simulation framework using OMNeT++, implement RELESS and its counterparts, and conduct simulation experiments for performance evaluation and comparison. Numerical results demonstrate that RELESS can improve service request satisfaction ratio, service request satisfaction latency, as well as data size satisfaction ratio, indicating a superior service scheduling approach in the IoD.

I. Introduction

The global COVID-19 pandemic affects every one of us in some way during the last two years. Even as ordinary business shut down, critical industries and facilities such as utilities, mass transit, telecommunications, and oil and gas production were under great pressure to maintain regular operations continuously. All of a sudden, drone technology became the ultimate tool to boost efficiency and accuracy of everyday operations to combat COVID-19. For example, drones are being used extensively for medicine and grocery deliveries, disinfectant spraying, temperature check, and warning citizens to wear masks [1]. Thus, we argue that the global COVID-19 pandemic became an inflection point for drone industry. In addition, as stated in the “Drone Technology and Global Mar-kets” report from BCC Publishing, the global drone market is estimated to be worth approximately $55 billion in 2025, with a compound annual growth rate (CAGR) of around 13% for the period of 2020–2025 [2]. With the support and promotion of other advanced technologies (e.g., Artificial Intelligence (AI) and fifth-generation (5G) mobile communications), we anticipate that the use of drone technology will be even more widespread during the post-pandemic period [3].