Distribution-Regularized Federated Learning on Non-IID Data

Federated learning (FL) has emerged as a popular machine learning paradigm recently. Compared with traditional distributed learning, its unique challenges mainly lie in communication efficiency and non-IID (heterogeneous data) problem. While the widely adopted framework FedAvg can reduce communication overhead significantly, its effectiveness on non-IID data still lacks exploration. In this paper, we study the non-IID problem of FL from the perspective of domain adaptation. We propose a distribution regularization for FL on non-IID data such that the discrepancy of data distributions between clients is reduced. To further reduce the communication cost, we devise two novel distributed learning algorithms, namely rFedAvg and rFedAvg+, for efficiently learning with the distribution regularization. More importantly, we theoretically establish their convergence for strongly convex objectives. Extensive experiments on 4 datasets with both CNN and LSTM as learning models verify the effectiveness and efficiency of the proposed algorithms.

I. Introduction

Federated learning (FL) [1]–[3] is a new distributed machine learning paradigm that collaboratively trains models among multiple clients while the raw training samples possessed by each client cannot be shared. Federated learning has a wide range of real-world applications. For example, a large number of smartphone users can jointly train accurate next-word prediction models (a.k.a cross-device FL) [4], whereas enterprises or hospitals that do not have enough data for learning can cooperate to train federated models under privacy regulations (a.k.a cross-silo FL) [3].