Leveraging Curriculum Reinforcement Learning for Rocket Powered Landing Guidance and Control

Future missions to recover the first stage of rocket require advanced guidance and control algorithms which can overcome stochastic disturbances, so as to achieve pinpoint landing. Recent studies have shown promising results of reinforcement learning (RL) based powered landing guidance and control method. However, many of them suffer from poor sample efficiency due to the sparse and complex reward, that trades off different goals in terms of attitude stability, terminal velocity, terminal attitude, terminal position and fuel usage. In order to address the above-mentioned problem, we propose a novel curriculum RL method to learn a fuel optimal pinpoint landing policy by maximizing the complex reward in this paper. The basic idea is to start with an easier task, and then increase the difficulty level by adding more goals. In this way, each task provides a good initial policy for the next task, making it easier to train the policy. The plane rigid dynamic model of rocket powered landing is adopted. The simulation result shows that the proposed RL method, which combined curriculum learning and Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm, can learn a better policy than the TD3 algorithm without curriculum under the same number of samples.