Multiple Subgoals-guided Hierarchical Learning in Robot Navigation

Solving obstacle-clustered robotic navigation tasks via model-free reinforcement learning (RL) is challenging due to the extended decision horizon and sparse rewards. Previous work has demonstrated efficient learning with single subgoal-conditioned hierarchical approaches. The subgoal is the action from the high-level policy and it operates on the low-level module, which could invoke sub-optimal policy when the selected subgoal is suboptimal. This work introduces multiple subgoals-guided navigation (MSGN) which consists of a high-level multiple subgoals Planner and a low-level goal-conditioned RL Controller. By passing multiple subgoals to the low-level agent, MSGN could alleviate the suboptimal subgoal problem by transferring the subgoal selection process to the RL agent. At the same time, multiple subgoals could help the goal-conditioned RL agent better explore and understand the environment and task. We tested our method on the Safety Gym suite. The results verified that MSGN could achieve a higher success rate and lower collision cost compared to baselines.