Synergetic Focal Loss for Imbalanced Classification in Federated XGBoost

Applying sparsity- and overfitting-aware eXtreme Gradient Boosting (XGBoost) for classification in federated learning allows many participants to train a series of trees collaboratively. Since various local multiclass distributions and global aggregation diversity, model performance plummets as convergence slowly and accuracy decreases. Worse still, neither the participants nor the server can detect this problem and make timely adjustments. In this article, we provide a new local-global class imbalance inconsistency quantification and utilize softmax as the activation and focal loss, a dynamically scaled cross-entropy loss, in federated XGBoost to mitigate local class imbalance. Moreover, we propose a simple but effective hyperparameter determination strategy based on local data distribution to adjust the sample weights among noncommunicating participants, synergetic focal loss, to solve the inconsistency of local and global class imbalance, a unique characteristic of federated learning. This strategy is perfectly integrated into the original classification algorithm. It requires no additional detectors or information transmission. Furthermore, a dynamical for loop is designed to capture an optimum hyperparameter combination. Finally, we conduct comprehensive tabular- and image-based experiments to show that synergetic focal loss used in federated XGBoost achieves faster convergency and significant accuracy improvement. Simulation results prove the effectiveness of the proposed principle of configuring sample weights.

I. Introduction

Federated learning (FL) [1] has become a popular distributed framework that enables tremendous participants who own local datasets cooperatively to train a joint machine learning (ML) model. FL essentially solves problems of client privacy [2], data and profit ownership [3], and takes advantage of edge computation [4] to a large extent. Classification methods play a crucial role in knowledge discovery of databases and data mining, such as sparse support vector machine [5], stochastic gradient descent (SGD) based deep learning [6], graph neural networks [7], gradient boosting decision trees (GBDTs) [8], and meta-learning [9] is widely used in FL. XGboost [10], with highly efficient, flexible, and portable characteristics, is famous for both classification and regression tasks. It was utilized in many Kaggle competitions winning teams and was the third most commonly used ML algorithm. In the foreseeable future, eXtreme Gradient Boosting (XGBoost) and its variants are still the most popular methods in the data science community.