Proximal Policy Optimization Algorithm for Enhancing Energy Harvesting in UAV-Assisted Communications with RIS

The integration of reconfigurable intelligent surfaces (RIS) into Unmanned Aerial Vehicles (UAVs) can be deployed to provide ubiquitous communication services in communication-impaired areas, but the limited on-board battery capacity of the UAVs restricts their endurance. The operating time of the UAV can be extended by dividing the metasurface elements on the RIS, but channel variations due to the constant movement of pedestrians present a significant challenge to efficient resource allocation. In this paper, a novel energy harvesting (EH) scheme, called the UAV-EH, is developed based on the UAV-RIS system. The proposed UAV-EH scheme extends the Synchronized Wireless Information and Power Transfer (SWIPT) system with simultaneous signal transmission and energy harvesting on the RIS metasurface. And we develop a robust proximal policy optimization (PPO) algorithm combined with cropping the action space to assign RIS metasurface elements as a way to improve the UAV-EH scheme to ensure quality of service (QoS) in dynamic wireless environments. The simulation results demonstrate the effectiveness of our proposed UAV-EH scheme based on PPO, with average energy harvesting rates of 65.7% for single-user terminals and 65.6% for multi-user terminals. These rates are very close to the performance of exhaustive search algorithms and outperform all other schemes.