Spiking Mean Field Multi-Agent Reinforcement Learning for Dynamic Resources Allocation in D2D Networks

Device-to-device (D2D) technology has been widely used to alleviate the mobile traffic explosion due to its ability of direct communications between proximal devices. However, in practice, available spectrums are limited and the number of D2D and cellular users are rapidly increasing, which greatly decreases the efficiency of resource allocation. For this propose, we train spiking neural network (SNN) with deep reinforcement learning for channel selection and power control. At the same time, we use spatio-temporal backpropagation to accelerate SNN training. When the number of D2D users increases dramatically, the learning rate becomes intractable due to the curse of the exponential growth of action space. Therefore, we apply mean field multi-agent reinforcement learning (MFRL) to approximate interactions within the D2D users. After that, we combine different reinforcement learning algorithms with MFRL. The simulation result shows that, compared with actor-critic (AC) and proximal policy optimization (PPO), spiking actor-critic (S-AC) and spiking proximal policy optimization (S-PPO) can achieve faster convergence rate, higher access rate and better time-averaged overall throughput as well as lower collision probability even when the action space is increased. Besides, when the number of D2D users increases, our spiking mean field proximal policy optimization (SMF- PPO) can achieve better performance than AC, PPO, S-AC and S-PPO.