Communication-Efficient Federated Multi-Task Learning with Sparse Sharing

Federated multi-task learning (FMTL) is a promising technology to deal with the severe data heterogeneity issue in federated learning (FL), where each client learns individual models locally and the server extracts similar model parameters from the tasks to keep personalization for models of clients. Hence, it is essential to precisely extract the model parameters shared among tasks. On the other aspect, the limitation of communication resources would also restrict the model transmission, and thus influence the FMTL performance. To address the above issues, we propose a novel FMTL with Sparse Sharing (FedSS) mechanism that allows clients to share model parameters dynamically according to diversified model structures under limited communication resources. Particularly, we present an adaptive quantization approach for task relevance, which serves as a metric to evaluate the extent of model sharing across tasks. The objective function is formulated to minimize the model transmission latency while ensure the FMTL learning performance via a joint bandwidth allocation and client selection strategy. Closed-form expressions for the optimal client selection and bandwidth allocation are derived based on a alternating direction method of multipliers (ADMM) algorithm. Numerical results show that the proposed FedSS outperforms the benchmarks, and achieves efficient communication performance.

I. Introduction

With the breakthrough of machine learning (ML), various intelligent services and applications have been witnessed a thriving development recently. However, the centralized ML method requires transmitting a large amount of data to the cloud for model training, which leads to significant resource consumption and privacy protection issues [1]. Hence, federated learning (FL) receives widespread attentions from both academia and industry. Particularly, FL can collaborate with multiple devices to train a shared ML model by transferring model parameters rather than large amounts of private data, thus effectively reducing resource consumption and enhancing privacy protection [2]. Despite of these benefits, FL still faces challenges in data heterogeneity [3], [4] and communication overhead [5], [6], which would significantly decrease the FL performance.