Adaptive Regularization and Resilient Estimation in Federated Learning

Federated Learning (FL) is an emerging research area that produces a globally trained model using numerous local users’ data and maintains their privacy. Heterogeneous or non-Independent and Identically Distributed (non-IID) data affect the global model's convergence and, therefore, cause high communication costs. These are because traditional FL approaches often disregard an adaptive regularized objective for the user-side training and utilize conventional arithmetic mean on the locally trained models for the server-side aggregation. To alleviate these issues, we propose a novel FL scheme in this paper. In particular, we propose an adaptive regularization approach to add to the classical objective function of the users’ local models during training and a resilient estimation approach to the locally trained models during aggregation. The adaptive regularization approach is derived using the users’ local and global performance diversification while the resilient estimation scheme uses a modified geometric mean aggregation over the local models’ parameters. We provide consolidated theoretical results and perform extensive experiments on the IID and non-IID settings of MNIST, CIFAR-10, and Shakespeare datasets with various deep networks. The results manifest that our FL scheme outperforms the state-of-the-art approaches in terms of communication speedup, test-set performance, training convergence stability, and resiliency against attacks.

I. Introduction

The rise in popularity of smartphones, tablets, and wearable users has resulted in an overwhelming of rich data, such as images taken and text inputted by users. On the one hand, there is a potential scope to improve user experience and empower intelligent programs (e.g., analysis of user's activity, analysis of user's sentiment, analysis of image content, clinical care analysis of diabetes and heart disease, burglary analysis in a smart residential area, etc.). On the other hand, it poses privacy concerns, where a user's personal information has to be kept concealed. In this scenario, the users can be prepared with additional data computations rather than direct sharing of the private raw data as they are usually built with large processing and storage resources. As such, Federated Learning (FL) has been recently introduced to meet this purpose [1], [2], [3], [4].