Personalized Federated Learning With Differential Privacy and Convergence Guarantee

Personalized federated learning (PFL), as a novel federated learning (FL) paradigm, is capable of generating personalized models for heterogenous clients. Combined with a meta-learning mechanism, PFL can further improve the convergence performance with few-shot training. However, meta-learning based PFL has two stages of gradient descent in each local training round, therefore posing a more serious challenge in information leakage. In this paper, we propose a differential privacy (DP) based PFL (DP-PFL) framework and analyze its convergence performance. Specifically, we first design a privacy budget allocation scheme for inner and outer update stages based on the Rényi DP composition theory. Then, we develop two convergence bounds for the proposed DP-PFL framework under convex and non-convex loss function assumptions, respectively. Our developed convergence bounds reveal that 1) there is an optimal size of the DP-PFL model that can achieve the best convergence performance for a given privacy level, and 2) there is an optimal tradeoff among the number of communication rounds, convergence performance and privacy budget. Evaluations on various real-life datasets demonstrate that our theoretical results are consistent with experimental results. The derived theoretical results can guide the design of various DP-PFL algorithms with configurable tradeoff requirements on the convergence performance and privacy levels.

I. Introduction

It is expected that big data-driven artificial intelligence (AI) will soon be applied in many aspects of our daily lives, including health care [1], communications [2], [3], autonomous driving [4], etc. At the same time, the rapid growth of Internet-of-Things (IoT) applications calls for data mining and model learning securely and reliably in distributed systems. In integrating AI in a variety of IoT applications, distributed machine learning (ML) systems are preferred for data processing tasks with edge intelligence [5]. Federated learning (FL) [6], [7], [8], as a recent advance in distributed ML, has been proposed to ensure data being processed locally, thereby protecting clients’ privacy.