The Impact of Dataset on Offline Reinforcement Learning Performance in UAV-Based Emergency Network Recovery Tasks

Emergency network recovery using unmanned aerial vehicles (UAVs) has gained significant attention in disaster scenarios. Offline reinforcement learning (RL) emerges as a promising solution for training UAVs, entailing the development of policies from pre-collected datasets. Numerous studies employ expert demonstration datasets to optimize the training performance of offline RL. Interestingly, we have empirically found that the expert demonstration dataset underperforms in UAV training scenarios due to the inherent task characteristic: sparse reward environments, wherein agents only attain rewards upon goal accomplishment. We delve deeper into the relationship between the dataset properties and offline RL performance. This outcome arises from the dataset’s interplay between reward ratio and trajectory diversity. We propose a framework to estimate offline RL performance by incorporating dataset diversity and the reward ratio using the trade-off between them. Finally, simulation results confirm that maintaining an appropriate balance of reward ratio and trajectory diversity during offline RL substantially enhances learning performance.