An Empirical Analysis of Rebalancing Methods for Security Issue Report Identification

Identifying security vulnerabilities in issue reports is a complex and time-sensitive task that when carried out effectively and in a timely manner can prevent attackers from exploiting software systems. While it is possible to address this using machine learning, the heavy imbalance of the datasets involved requires a meticulous use of rebalancing methods to achieve reasonably effective models. In this paper we analyze the effectiveness of different data rebalancing methods (e.g., oversampling and undersampling) applied to the classification of security issue reports using machine learning techniques. Our results using the Ubuntu dataset show that oversampling is an overall better strategy for rebalancing, SVMSMOTE and Random Undersampling are the individual methods that show the best performance. We also found that variations in the proportion of the minority class have little effect on the difference in the effectiveness of the best methods. Overall, our results are useful for creating more effective machine learning models for the automatic identification of security bug reports.