Heterogeneous Observation Aggregation Network for Multi-agent Reinforcement Learning

Learning effective policies is challenging for a multi-agent system in partially observable environments, where agents need to extract relevant features from local observations. Most approaches in multi-agent reinforcement learning (MARL) are limited to feature extraction for homogenous agents. They struggle to deal with local observations in heterogeneous multi-agent scenarios, where agents have different observation spaces and are necessitated to process semantically varied information. To address this issue, we analyze the observational heterogeneity of multi-agent systems, and propose a heterogeneous-graph-based approach for feature extraction in MARL. We model agent observations as heterogeneous graphs, and design a heterogeneous observation aggregation network (HOA-Net) for processing these graph-based observations. HOA-Net is specifically designed to address various forms of observational heterogeneity. It employs class-specific weighting networks and computes across-class attentions for observed entities, effectively reducing the number of learnable parameters. The proposed method is evaluated on SMAC and an Unreal-Engine-based heterogeneous multi-agent testbed. Experimental results demonstrate that our method significantly outperforms other baselines in effectively aggregating an agent’s observation, and finally enhancing the performance of heterogeneous multi-agent systems.