Model Based Reinforcement Learning Pre-Trained with Various State Data

Reinforcement Learning (RL) has shown remarkable capabilities in various domains, yet struggles in environments with sparse rewards. A significant challenge in such environments is the exploration depth and the robustness of performance. This paper introduces WODID framework, aiming to enhance exploration in Model-Based Reinforcement Learning (MBRL) without relying heavily on initial or early-stage trajectory data. We identify one primary issue of the transition model of MBRL: trained with random policy when forming the transition model, which hinders exploration and causes high dependency on the success of dataset collection by random policy. By pre-training world models using diverse state data, WODID improves the quality of the transition model, leading to deeper exploration and stabilizing its performance. Our empirical studies, particularly in the challenging sparse reward environment: Montezuma’s Revenge, demonstrate that WODID outperforms the baseline methods, achieving more profound exploration with fewer environmental steps. Furthermore, our approach offers a human-free method to feed trajectory data, promoting less dependency on initial samples and paving the way for more robust and efficient RL agents.