On the Convergence of Multi-Server Federated Learning With Overlapping Area

Multi-server Federated learning (FL) has been considered as a promising solution to address the limited communication resource problem of single-server FL. We consider a typical multi-server FL architecture, where the coverage areas of regional servers may overlap. The key point of this architecture is that the clients located in the overlapping areas update their local models based on the average model of all accessible regional models, which enables indirect model sharing among different regional servers. Due to the complicated network topology, the convergence analysis is much more challenging than in single-server FL. In this paper, we firstly propose a novel MS-FedAvg algorithm for this multi-server FL architecture and analyze its convergence on non-iid datasets for general non-convex settings. Since the number of clients located in each regional server is much less than single-server FL, the bandwidth of each client should be large enough to successfully communicate training models with the server, which indicates that full client participation can work in multi-server FL. Also, we provide the convergence analysis of the partial client participation scheme and develop a new biased partial participation strategy to further accelerate convergence. Our results indicate that the convergence results highly depend on the ratio of the number of clients in each area type to the total number of clients in all three strategies. The extensive experiments show remarkable performance and support our theoretical results.

1 Introduction

With the explosive growth in the numbers of mobile phones and Internet of Things (IoT) devices, a tremendous amount of data today is being generated at the network edge in a distributed manner. Sending this data to the cloud for processing not only puts a huge burden on the network but also raises serious data privacy concerns. Federated Learning (FL) [1], [2], [3] recently emerged as a distributed Machine Learning (ML) architecture that keeps all the training data on individual clients, thereby protecting client data privacy and mitigating network congestion.