Two-Phase Deep Reinforcement Learning of Dynamic Resource Allocation and Client Selection for Hierarchical Federated Learning

This paper presents a new two-phase Deep Deter-ministic Policy Gradient (DDPG) framework, referred to as “TP-DDPG”, to balance online the learning delay and model accuracy of a federated learning (FL) process in an energy harvesting hierarchical FL (HFL) system. The key idea is to design a DDPG-based approach to learn the selection of participating clients, the CPU configuration, and the transmission powers of the clients, while the other decisions are efficiently optimized by a new straggler-aware client association and bandwidth allocation algorithm. The algorithm evaluates the reward of the DDPG, and substantially improves its convergence rate and stability. Experimental results demonstrate that the proposed TP-DDPG can substantially reduce the training time while achieving a higher test accuracy over the existing benchmarks.