OH-DRL: An AoI-Guaranteed Energy-Efficient Approach for UAV-Assisted IoT Data Collection

In this paper, we propose a hierarchical optimization approach that guarantees the maximum age of information (AoI) for unmanned aerial vehicle (UAV) assisted Internet-of-Things (IoT) data collection. Our model is based on an energy-efficient simultaneously transmitting and reflecting reconfigurable intelligent surface (STAR-RIS) beamforming model. We formulate the optimization to minimize the UAV flight energy consumption subject to a maximum average AoI threshold by optimizing the UAV trajectory, IoT device scheduling, and STAR-RIS beamforming. To solve this, we develop an optimization-based hierarchical deep reinforcement learning (OH-DRL) algorithm that decomposes the formulated problem into an inter-cluster UAV visiting policy and STAR-RIS-based intra-cluster IoT scheduling policy. In OH-DRL, we jointly optimize the two policies in a high-level loop and a low-level loop, respectively. In the high-level loop, we design an AoI-guided DRL algorithm to determine the AoI-guaranteed UAV hovering position with minimal flight distance. In the low-level loop, a semidefinite relaxation (SDR)-based optimization algorithm further reduces the UAV’s flying time by minimizing the average AoI. Simulation results validate that OH-DRL achieves better convergence performance and energy-saving efficiency across different network scales. Compared to the state-of-the-art DRL algorithm, OH-DRL reduces the UAV flight energy consumption by 14.4% and decreases the number of training episodes required for convergence by 66%.