Federated Continual Learning via Knowledge Fusion: A Survey

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called “spatial-temporal catastrophic forgetting” and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.

I. Introduction

Deep learning has been instrumental in the advances of both data analysis and artificial intelligence (AI) [1]. The algorithms have also been successfully used in almost all areas of applications in industry, science, and engineering [2]. However, traditional deep learning algorithms often require large amounts of training data and centralized training, which are usually impractical or even impossible in practical situations. For example, in the case of mobile devices, it is difficult to collect and transfer large amounts of data to a central server with limited bandwidth [3]. Centralized training is also infeasible due to data privacy and data silos because of the information security, related laws and regulations, and intensive competition among technology giants [4]. In recent years, federated learning (FL) has emerged as a positive response to the increasing need for privacy and other concerns in machine learning applications [5]. This newly emerging paradigm deploys multiple devices to train a global model collaboratively by uploading their models to a server while keeping the private data locally, as opposed to the traditional centralized data storage and model training approach. FL mainly involves privacy preservation, knowledge transfer, and collaboration among data owners, which are essential for many real-world applications of AI. After several years of development, FL has produced promising results, particularly in industries that handle sensitive data such as finance, healthcare, electronic communication and government affairs [6], [7], [8].