Quadratic Exponential Decrease Roll-Back: An Efficient Gradient Update Mechanism in Proximal Policy Optimization

Proximal Policy Optimization (PPO) achieved the best performance in many continuous control tasks as a popular Deep reinforcement learning algorithm. However, the maximum point of the surrogate in the PPO and its variant version is not the zero point of the policy gradient. In this paper, we propose a variant of PPO: PPO with Quadratic Exponential Decrease roll-back (PPO-QED), which uses a quadratic, exponential decrease roll-back function to make the zero point and the maximum point coincide. We prove that the policy gradient of the PPO-QED is bounded and that the policy gradient of PPO-QED can approach the zero point more steadily than PPO-RB. We experiment with the proposed algorithm in continuous control tasks, selecting PPO-KL, PPO-Clip, PPO-RB, PPO-S, and TRGPPO as a comparison; the experimental results show that the PPO-EDRB performs well.