Loci: Federated Continual Learning of Heterogeneous Tasks at Edge

Federated continual learning (FCL) has attracted growing attention in achieving collaborative model training among edge clients, each of which learns its local model for a sequence of tasks. Most existing FCL approaches aggregate clients’ latest local models to exchange knowledge. This unfortunately deviates from real-world scenarios where each model is optimized independently using the client’s own dynamic data and different clients have heterogeneous tasks. These tasks not only have distinct class labels (e.g., animals or vehicles) but also differ in input feature distributions. The aggregated model thus often shifts to a higher loss value and incurs accuracy degradation. In this article, we depart from the model-grained view of aggregation and transform it into multiple task-grained aggregations. Each aggregation allows a client to learn from other clients to improve its model accuracy on one task. To this end, we propose Loci to provide abstractions for clients’ past and peer task knowledge using compact model weights, and develop a communication-efficient approach to train each client’s local model by exchanging its tasks’ knowledge with the most accuracy relevant one from other clients. Through its general-purpose API, Loci can be used to provide efficient on-device training for existing deep learning applications of graph, image, nature language processing, and multimodal data. Using extensive comparative evaluations, we show Loci improves the model accuracy by 32.48% without increasing training time, reduces communication cost by 83.6%, and achieves more improvements when scale (task/client number) increases.

I. Introduction

Artificial intelligence (AI) applications are widely deployed on edge devices or clients, e.g., classifying image objects [1], [2], translating text [3], and recognizing patterns in social graphs [4]. Federated learning (FL) [5] enables edge clients to collaboratively learn a global model by aggregating their local models without exchanging the raw data. Federated continuous learning (FCL) addresses scenarios where clients incrementally train models over a sequence of tasks characterized by their data distributions. For example, Fig. 1(a) illustrates a FCL scenario of $n$ edge clients in an image classification application. Each client learns its own sequence of heterogeneous tasks, namely distinct class labels (e.g., cars, ships, and houses in client 1, and cats, dogs, and horses in client 2) and a subspace of input feature distribution for each task. Fig. 1(c) further shows that such heterogeneous tasks widely exist in edge AI applications of different data modalities, such as graph [4], text [3], and multi-modal data [1], [2]. It is exceedingly challenging to learn AI models that can be generalized on such heterogeneous tasks, which vary across clients and over time, especially on resource strenuous edge devices.