Differentially-Private Federated Learning with Long-Term Constraints Using Online Mirror Descent

This paper discusses a fully decentralized online federated learning setting with long-term constraints. The fully decentralized setting removes communication and computational bottlenecks associated with a central server communicating with a large number of clients. Also, online learning is introduced to the federated learning setting to capture situations with time-varying data distribution. Practical federated learning settings are imposed with long-term constraints such as energy constraints, money cost constraints, time constraints etc. The clients are not obligated to satisfy any per round constraint, but they must satisfy these long-term constraints. To provide privacy to the shared local model updates of the clients, local differential privacy is introduced. An online mirror descent-based algorithm is proposed and its regret bound is obtained. The regret bound is compared with the regret bound of a differentially-private version of online gradient descent algorithm proposed for federated learning.

I. Introduction

Federated learning is a distributed learning paradigm in machine learning, where clients train a shared model in a collaborative way while preserving their local data [1]. The original model for federated learning includes a central server that does the coordination. The central server serves as the master, while the clients serve as the workers. The clients only send their model updates to the server at each iteration. Hence, the central server has no access to the clients' local data. The central server aggregates the updates received from all the clients at each iteration, and broadcasts a global model back to all the clients. This continues till the optimal global model is reached. Federated learning has been successfully applied for next word prediction in natural language processing [2], emoji prediction [3], vocabulary estimation [4], vehicle-to-vehicle and wireless communication [5]–[7], social networks [8]–[10] and predictions in health [11], [12].