Multi-UAV Speed Control with Collision Avoidance and Handover-Aware Cell Association: DRL with Action Branching

This paper develops a deep reinforcement learning solution to simultaneously optimize the multi-UAV cell-association decisions and their moving velocity decisions on a given 3D aerial highway. The objective is to improve both the transportation and communication performances, e.g., collisions, connectivity, and HOs. We cast this problem as a Markov decision process (MDP) where the UAVs' states are defined based on their velocities and communication data rates. We have a 2D transportation-communication action space with decisions like UAV acceleration/deceleration, lane-changes, and UAV-base station (BS) assignments for a given UAV's state. To deal with the multi-dimensional action space, we propose a neural architecture having a shared decision module with multiple network branches, one for each action dimension. A linear increase of the number of network outputs with the number of degrees of freedom can be achieved by allowing a level of independence for each individual action dimension. To illustrate the approach, we develop Branching Dueling Q-Network (BDQ) and Branching Dueling Double Deep Q-Network (Dueling DDQN). Simulation results demonstrate the efficacy of the proposed approach, i.e., 18.32% improvement compared to the existing benchmarks.

I. Introduction

Unmanned aerial vehicles (UAVs) are gaining popularity across a broad range of applications due to their mobility, flexible deployment, gradually decreasing production costs, and line-of-sight (LOS) channels. [1]. A UAV can either require cellular connectivity for its own use (UAV-UEs) or provide cellular coverage as a base station (BS). Nevertheless, controlling UAVs that operate beyond visual line of sight (BVLoS) requires reliable command and control which is crucial for mission safety and security.