Proximal policy optimization with proportional-differential feedback for tracking control of unmanned surface vessel

Reinforcement learning has proven effective in guiding the motion of unmanned surface vessel (USV) without the need for intricate models or precise computations. In this paper, we propose a proximal policy optimization with proportional-differential feedback (PPO- PD) algorithm, building upon the proximal policy optimization (PPO) framework, to address the USV trajectory tracking control challenge. Drawing inspiration from the stability augmentation system (SAS) concept, we introduce a proportional-differential (PD) feedback controller into the environment prior to PPO controller training. This collaborative effort between PPO and PD optimizes the control strategy and enhances system convergence. Furthermore, a sim-ulation environment is set up based on the kinematic model of the USV. Experimental simulations demonstrate that the PPO-PD controller exhibits commendable performance in terms of tracking stability and robustness.