Energy Efficient RIS-Assisted UAV Networks Using Twin Delayed DDPG Technique

Unmanned Aerial Vehicle (UAV) has emerged as a promising technology to provide wireless signals from air to the ground users in specific scenarios such as earthquakes, tsunamis and other disasters. The performance of the UAV is degraded when the signals are blocked by obstacles in dense urban scenarios. To address this issue and enhance the signal quality available to the ground users, Reconfigurable Intelligent Surface (RIS) has emerged as a new technological paradigm. It offers an intelligent configuration for the signal propagation environment by redirecting the signals to the users. In this article, we solve a non-convex optimization problem of RIS-assisted UAV network by jointly optimizing the RIS phase shift and 3D trajectory of UAV to maximize the energy efficiency of a rotatory-wing UAV. The considered optimization problem is solved using Deep Reinforcement Learning (DRL) based techniques in an on-line fashion to reduce the computational complexity. We leverage Twin-delayed Deep Deterministic Policy Gradient (TD3) to solve the problem by considering the UAV trajectory as a set of continuous actions. For comparison, we also use the Soft Actor-Critic (SAC), Deep Deterministic Policy Gradient (DDPG) and Double Deep Q-Network (DDQN) for continuous and discrete optimization of the UAV trajectory, respectively. Extensive simulations show that the TD3 outperforms all the considered DRL techniques with the highest energy efficiency and throughput, and the lowest propulsion energy.

I. Introduction

Unmanned Aerial Vehicle (UAV) has been used as one of the important technologies in contemporary wireless networks design. Thanks to their low cost, 3D-position flexibility, and real-time service provision, the use of UAVs can be a promising approach in up-coming 6G communication. They can be used to provide efficient and real-time connectivity to the users in various applications such as disasters recovery, vehicle to-everything (V2X) communications, smart city development, surveillance and traffic offloading in hot spots [1]. With Line-of-Sight (LOS) links, the UAVs provide the on-demand communications to the ground users of different critical services with Quality of Service (QoS) requirements [2]. However, the mobility and the trajectory management of UAV are major challenges of UAV networks [3]. Some other challenges in UAV-assisted communications include Energy Efficiency (EE), real-time data transmission, channel access, security protection and intelligent learning [4].