Efficient Off-policy Adversarial Imitation Learning with Imperfect Demonstrations

Generative Adversarial Imitation learning (GAIL) enables the agent to learn a desired policy from the demonstrations provided by human beings. For complex continuous control tasks, such as robotic tasks, demonstrations are usually be obtained from multiple demonstrators to save time. However, the differences in the abilities of multiple demonstrators lead to the drift of demonstration distribution, which results in learning a suboptimal policy in the adversarial training process. In this paper, we proposes a new imitation learning (IL) algorithm, IM-GAIL, to enable the agent to learn effectively from imperfect demonstrations of multiple demonstrators. The proposed method rebuild the objective function in the adversarial training, making the policy match the optimal policy. Meanwhile, to improve the sample efficiency, off-policy training framework is adopted and a curious experience replay mechanism is designed for the training. The experiments are conducted on robotic simulation tasks, and results show that our method can efficiently learn from the given demonstrations and it achieves better performance than other state-of-the-art methods.