A Brief Review of Recent Hierarchical Reinforcement Learning for Robotic Manipulation

Deep reinforcement learning (DRL) has become a popular learning paradigm for decision and control and has been widely applied in robot manipulation in recent years. However, due to its special learning pattern of “trial and error”, there are still some remaining problems with DRL. Such as exploration dilemma, sample inefficient, and slow convergence, are to be refined, especially when faced with complex long-horizon tasks. As a solution for these limits, hierarchical reinforcement learning (HRL) is proposed and developed by decomposing challenging tasks into multiple simpler subtasks, to efficiently solve the main task in a“divide and conquer” manner. At present, there are comprehensive HRL methods for robotic manipulation tasks, while a review is lacking. To facilitate researchers to form a general view of this field, we systematically summarize related HRL methods for robotic manipulation. This review carries out literature sortation in a novel taxonomy of subtask generation and divides HRL methods into two categories: handcrafted subtask generation and learning-based subtask generation. A great number of representative methods are analyzed in detail. In the end, we also present some important future directions for HRL.