Research on Imbalanced Data Ensemble Algorithm Based on Density Peaks Clustering and Fitness

Data imbalance is the phenomenon that there is a large difference in the number of positive and negative samples in the index data set. The traditional classification algorithms have poor performance in classifying positive samples on unbalanced data sets, which affects people's data mining analysis of interesting samples. Today, the problem of data imbalance is becoming more and more common, and there are many application areas of data imbalance, such as medical diagnostics, fraud detection and target detection. For this reason, relevant researchers have proposed many methods to solve the problem of data imbalance, the main methods include oversampling, undersampling, ensemble learning, threshold shifting. The existing methods of data resampling combined with ensemble learning tend to lose most of the important class samples, which leads to poor learning effect of the classifier, and further affects the classification accuracy of unbalanced data. To solve this problem, an unbalanced data integration algorithm DPFBoost based on density peak clustering and fitness is proposed in this paper. In addition, in order to shorten the clustering time of the density peak clustering algorithm used by DPFBoost on large data sets and improve its application universality, this paper optimized the algorithm based on Spark and proposed a SparkDPC parallel algorithm based on Spark. The overall contents of the study are as follows:(1) An unbalanced data integration algorithm DPFBoost based on density peak clustering and fitness is proposed. The main method is to cluster most class samples in unbalanced data sets and calculate the number of under-samples generated by class clusters. The number of under-samples of different class clusters is related to the density of such clusters. Then, the local density of samples in the cluster is taken as the fitness of samples, and each cluster is undersampled respectively. Finally, combined with ensemble learning, multiple resampling and iterative training are used to improve the classification performance of unbalanced data. The classification accuracy of DPFBoost algorithm for unbalanced data sets is higher than other comparison algorithms, which shows that the proposed algorithm has better classification performance for unbalanced data sets.(2) SparkDPC mainly uses density peak clustering algorithm to calculate Euclidean distance between samples and local density of sample points for parallel design. Through analysis, it can be seen that the time complexity of calculating Euclidean distance and local density by density peak clustering algorithm is O(n2). When the amount of data is large, the calculation of these two steps will cause a lot of time overhead. Therefore, this paper designs these two steps in parallel based on Spark to improve the efficiency of the algorithm. When SparkDPC runs on a Spark cluster consisting of multiple nodes, the acceleration ratio of the algorithm increases with the increase in data volume. Therefore, the clustering efficiency of the algorithm is higher, indicating that SparkDPC is more suitable for clustering large-scale data sets.