Reinforcement Learning with Neighborhood Search and Self-learned Rules

Reinforcement learning (RL), induced by the exploration-exploitation dilemma, usually requires enough interaction experiences with the environment to improve performance. Rule-based methods utilize the internal expertise of tasks to benefit the learning process of the target task and can be shared with similar goal-setting tasks, which is also one form of knowledge transfer with better interpretability. However, symbolic rules or prior knowledge of the task may not be obtained in advance for most control tasks. In this paper, we propose an effective approach named neighborhood search and self-learned rules (RL-NSSR), where the neighborhood search library and the rules can be learned from the samples of a set of collected trajectories. In the RL- NSSR, first, the expert policy is obtained by training the RL agent in the discretized continuous state-action space task for collecting the trajectories data. Second, we extract the rules using the collected data via ensemble decision tree methods, and select the states of the trajectories with higher cumulative rewards as the neighborhood states to speed up the learning process. Third, we employ both the neighborhood search and the self-learned symbolic rules in a fine-grained way and a rough way, respectively, to guide the exploration strategy for accelerating the learning process. Experiments conducted on several groups of locomotion tasks by the MuJoCo physics engine demonstrate that the proposed RL-NSSR approach can achieve improved control performances for both simple and complex tasks.