Large-scale multi-agent reinforcement learning using image-based state representation

With its high-dimensional state and action space, large-scale multi-agent reinforcement learning (MARL) is a challenging problem. Centralized approximate RL is impractical to deal with this because the search cost grows exponentially with the number of agents. Further, traditional decentralized approaches require delicate model-specific decomposition and communication within multi-agent system (MAS). Recently, however, convolutional neural network (CNN) has been successfully integrated into perceptional RL with high-dimensional input images. If the information of a MAS can also be recorded by a stack of images, the spatial correlation would be naturally represented by `pixel' positions and the kernel filter that allows for the multi-scale communications in MAS. Based on this observation, this paper proposes a new image-based representation for MARL by combining CNN for feature extraction and linear regression for Q-function approximation. Additionally, we perform decentralized search for the optimal action based on the structure of the learning model, and then propose a distributed fitted Q-iteration framework for general-purpose MARL. Initial experiments demonstrate that this CNN-enhanced RL agent has competitive performance when compared to the traditional ones, which learn from a manually designed flat feature representation. These encouraging results demonstrate that the image-based representation can provide promising opportunities to represent and analyze large-scale MARL using deep learning techniques.