Learning a Single Policy for Diverse Behaviors on a Quadrupedal Robot Using Scalable Motion Imitation

Learning various motor skills for quadrupedal robots is a challenging problem that requires careful design of task-specific mathematical models or reward descriptions. In this work, we propose to learn a single capable policy using deep reinforcement learning by imitating a large number of reference motions, including walking, turning, pacing, jumping, sitting, and lying. On top of the existing motion imitation framework, we first carefully design the observation space, the action space, and the reward function to improve the scalability of the learning as well as the robustness of the final policy. In addition, we adopt a novel adaptive motion sampling (AMS) method, which maintains a balance between successful and unsuccessful behaviors. This technique allows the learning algorithm to focus on challenging motor skills and avoid catastrophic forgetting. We demonstrate that the learned policy can exhibit diverse behaviors in simulation by successfully tracking both the training dataset and out-of-distribution trajectories. We also validate the importance of the proposed learning formulation and the adaptive motion sampling scheme by conducting experiments.

i. Introduction

Quadrupedal robots can achieve various autonomous mis-sions by overcoming rough terrains that wheeled robots cannot traverse, but the control is not straightforward due to its high-dimensional state space and under-actuated dynam-ics. Roboticists have studied various approaches for legged robot control, ranging from model-based control [1]–[5] to learning-based approaches [6]–[9], which have demonstrated impressive agility and robustness on various quadrupedal robots. However, most of the prior works have focused on one given specific task, such as robust walking, running, or jumping, because they require very different control strate-gies. These task-specific controllers often require manual engineering based on the expert's prior knowledge, which can be either developing mathematical models for model-based controllers or shaping reward functions for learning-based algorithms. It requires even more effort if the developer wants to improve the naturalness of the behavior.