CMBE: Curiosity-driven Model-Based Exploration for Multi-Agent Reinforcement Learning in Sparse Reward Settings

Sparse rewards in reinforcement learning have long been a central research challenge, often tackled through various exploration methods. However, in multi-agent scenarios with larger state spaces and action spaces, sparser rewards, and more complex learning strategies, conventional exploration methods cannot achieve satisfactory results. In this paper, we introduce a novel exploration approach for multi-agent reinforcement learning that combines the strengths of model-based techniques and curiosity-driven methods. We use curiosity-driven exploration in the early stages to facilitate comprehensive exploration of the entire state space. As training progresses, the forward model, trained during the initial exploration phase, is employed to selectively explore crucial actions, allowing agents to discover effective strategies. Due to the decrease in the loss of the curiosity-driven method with an increase in the number of state visits and the increasing accuracy of the forward model with more state visits, we utilize the curiosity-driven loss as a measure of the uncertainty in the forward model. Subsequently, based on the magnitude of this uncertainty, we determine which intrinsic reward to employ. Through experiments in a sparse-reward SMAC environment, we demonstrate the effectiveness of our algorithm. Visualizations of the results further validate the efficacy of our approach in enhancing exploration.