Work Together to Keep Fresh: Hierarchical Learning for UAVs-assisted Data Time-Sensitive IoT

In the context of disaster warning and monitoring within the Internet of Things (IoT), the utilization of unmanned aerial vehicles (UAVs) as relays to gather time-sensitive data from disaster monitoring sensors and transmit it to the base station (BS) has emerged as a highly promising application. In UAV-assisted Data Time-Sensitive IoT (DTIoT), the Age of Information (AoI) serves as a critical performance metric that quantifies the timeliness of data collection, specifically referring to the duration it takes for data to travel from the sensor to the BS. Controlling the flight trajectories of multiple UAVs to minimize AoI is a challenge under energy constraints. Existing work typically uses deep reinforcement learning (DRL) algorithms to address UAV trajectory control problems. However, the task of controlling multi-agent continuous trajectories in complex DTIoT network states is hindered by sparse rewards, posing challenges in training deep neural network-based control policies using standard DRL methods. In this paper, we propose a network-oriented hierarchical reinforcement learning (NO-HRL) algorithm to control the UAVs’ flight trajectory in DTIoT networks for minimizing the AoI. We design the control policy based on a two-layer hierarchical DRL, where the upper layer selects the target and the lower layer executes it. We further propose a decoupled sequential training scheme for effectively training the mutually coupled two-layer DRL network of NO-HRL. The experiment results show that our algorithm outperforms other baselines in AoI optimization for DTIoT.

I. Introduction

The sixth generation (6G) wireless communication network aims to revolutionize user services and applications through the Internet of Things (IoT) [1], [2]. With the rapid development of unmanned aerial vehicle (UAV) technology, UAVs are widely used as air base station (BS) or relay [3] in data collection due to flexibly deploying and establishing line of sight (LoS) connections with wide-area IoT devices [4], [5]. As shown in Fig. 1, the UAV-assisted data time-sensitive IoT network (UAV-DTIoT) is an emerging paradigm, which has higher requirements for the reliability and freshness of data in ultra-reliable and low-latency communication services [6], [7]. In the disaster monitoring scenario discussed in this paper, UAVs serve as relays to timely update a large amount of time-sensitive data generated by sensors to the BS [8], [9]. In order to quantify the timeliness in time-sensitive tasks, age-of-information (AoI) has been introduced [10], [11]. AoI measures the overall performance of the DTIoT network, which requires the joint optimization of all UAV controls in the network. In disaster monitoring scenarios, AoI dynamically changes with the environment, making optimization more challenging.