Byzantine-Robust Compressed and Momentum-based Variance Reduction in Federated Learning

Federated learning involves a group of workers and a central server to train a machine learning model in a distributed manner. However, the distributed structure poses challenges in terms of communication efficiency and robustness against malicious attacks. Our work assumes the presence of Byzantine attackers among workers and considers Byzantine robustness in federated learning. In order to overcome the challenges of communication and Byzantine attacks, we propose a novel method called Byzantine-robust compressed and momentum-based variance reduction (BR-CoMVaR). This method innovatively combines Byzantine-robustness aggregation, gradient difference compression, and momentum-based variance reduction. Gradient difference compression helps to mitigate the effect of compression noise, while momentum-based variance reduction reduces the variance in local stochastic gradients and accelerates convergence. The incorporation of these two techniques with Byzantine-robustness aggregation further enhances the defense capability against Byzantine attacks and improves overall Byzantine robustness. Moreover, the extensive experimental results display the superior performance of BR-CoMVaR compared with other three most related algorithms.