A New Data-Free Knowledge Distillation Method for Non-IID Federated Learning

Federated Learning (FL) is an emerging distributed machine learning approach. It enables a server to aggregate model parameters from local users iteratively without requiring access to original data, thus maintaining data privacy. A persistent challenge in FL is the heterogeneity of data, which can slow convergence and impair the performance of the aggregated model. Addressing this, recent advancements in data-free knowledge distillation have shown promise. However, existing techniques often necessitate the transfer of private label distributions to the server, limiting their practical application. Therefore, this paper presents an innovative approach based on data-free knowledge distillation where generator and global model will be trained against each other. The generator is tasked with producing samples that are indistinguishable by the global model, fostering knowledge transfer from local to global models. Concurrently, the global model learns to replicate the outputs of the local models. This study also introduces a feature matching loss that enhances the training of the generator. By evaluating on various datasets, the proposed method shows a substantial performance improvement compared to previous methods.