Dynamically Interrupting Deadlocks in Game Learning Using Multisampling Multiarmed Bandits

In many reinforcement learning (RL) game tasks, an episode should be interrupted after a certain time, as the agent could sometimes fall into a deadlock state. The learning process is sensitive to the interruption length. So, it is hard to determine an optimal value of interruption. This article presents a novel multiarmed bandit (MAB) model to dynamically interrupt the deadlock state in reinforcement game learning, with the assumption that there are neither prior knowledge of optimal interruption setting, nor prior knowledge that can be used to improve the performance of the RL agent. The proposed MAB model is a nonoblivious adversarial MAB problem with a multisampling process in each round. An efficient algorithm named Exp3.P.MS is proposed for the new bandit setting, achieving an expected regret bound of
$\mathcal {O}(\sqrt{nK\ln {(K)}})$
. We run the algorithm on
Sokoban
. The experimental results show that the dynamic interruptions can adapt to the weak-to-strong performance of the RL agent and spur a fast learning in game training.