Computationally Efficient Imitation Learning via K-Timestep Adaptive Action Approximation

A key challenge for training control policies with imitation learning methods lies in the computational inefficiency. This inefficiency comes from an assumption that underlies these methods, assuming the policy should compute a new action for each state, which is unnecessary and costly. However, we notice the states occurring within K consecutive timesteps differ negligibly and their corresponding actions are extremely similar. Therefore, we challenge this assumption and argue that it is enough to compute an action every K states. With this argument, we propose K-Timestep Adaptive Action Approximation, which replaces the computation of K one-timestep actions approximately with that of one K-timestep action to alleviate the computational inefficiency issue. To demonstrate the theoretical validity of our method, we analyze the errors incurred by the policies learned via the method. The analysis proves these policies can converge to the optimal solution with errors no more than an upper bound dependent on K, revealing the effectiveness of our method. To avoid the difficulty of hyperparameter handcrafting on K, we design a simple but effective auto-hyperparameter tuning strategy. In the proposed strategy, K is added as an extra dimension to the action space of the policy, so it can be tuned adaptively by the policy according to the state without any user intervention. Empirical results on 4 imitation learning tasks show the superiority in computational efficiency of our method, which can effectively reduce new actions to compute in training policies.