Cost-Sensitive Ensemble Learning for Highly Imbalanced Classification

There are a variety of data-level and algorithm-level methods available for treating class imbalance. Data-level methods include data sampling strategies that pre-process training data to reduce levels of class imbalance. Algorithm-level methods modify the learning and inference processes to reduce bias towards the majority class. This study evaluates both data-level and algorithm-level methods for class imbalance using a highly imbalanced healthcare fraud data set. We approach the problem from a cost-sensitive learning perspective, and demonstrate how these direct and indirect cost-sensitive methods can be implemented using a common cost matrix. For each method, a wide range of costs are evaluated using three popular ensemble learning algorithms. Initial results show that random undersampling (RUS) and class weighting are both effective ways to improve classification when the default classification threshold is used. Further analysis using the area under the precision-recall curve, however, shows that both RUS and class weighting actually decrease the discriminative power of these learners. Through multiple complementary performance metrics and confidence interval analysis, we find that the best model performance is consistently obtained when RUS and class weighting are not applied, but when output thresholding is used to maximize the confusion matrix instead. Our contributions include various recommendations related to implementing cost-sensitive ensemble learning and effective model evaluation, as well as empirical evidence that contradicts popular beliefs about learning from imbalanced data.