Online-S2T: A Lightweight Distributed Online Reinforcement Learning Training Framework For Resource-Constrained Devices

In deep reinforcement learning (DRL), it is generally resource-constrained devices that interact with real-time dynamic environments, and many DRL models are computationally complex, making online training challenging. Previous research has proposed policy distillation to compress the trained large models into small models thus satisfying the requirements of resource-constrained devices. However, this is usually offline and cannot be suitable for real-time dynamic environments. In this work, we propose an online student-teacher (online-S2T) distributed framework which can deploy multiple smaller student models to resource-constrained devices for online training. Specifically, we decouple the training and interaction of the teacher models and enable the student models to interact with the environment instead of the teacher models. Moreover, to reduce the communication overhead between multiple student models and the teacher model and speed up model convergence, we propose the key experience extraction algorithm. It obtains representative sets of experiences from numerous experience trajectories. The evaluation shows that our approach further accelerates model training and improves the performance of converged models by 25.12% on average, with the ability to deploy smaller models on resource-constrained devices for training.