SPIL: Fast Imitation Learning with Single-Person Game

Imitation learning(IL) is a promising route to train robot control policies. Recent work applies a generative adversarial framework to imitation learning and achieves great success. However, these methods are rather expensive to train and require solving a difficult two-person game. We propose a novel IL method, SPIL, which replaces the two-person game algorithm with a single-person game. To concretely, our method avoids training a discriminator and sharply reduces learning time. We demonstrate that the common structure, discriminator in GAIL can be simplified to a simple reward function, which incurs an IL algorithm cheaper to solve. We carefully prove SPIL in theory and propose an easy approach to solve it. Experiments show that our method achieves equal accuracy to the state-of-the-art, while at a higher speed to converge.