Evaluating Bug Prediction under Realistic Settings

Bug prediction is expected to reduce the cost of quality assurance. To build a reliable bug prediction model, we should use realistic settings that satisfy all three of the following conditions. (1) We should build a dataset in a way that allows us to evaluate the prediction performance of the model correctly. (2) We should adopt the optimal granularity of bug prediction to minimize the cost of quality assurance. (3) We should use a dependent variable that correctly represents the presence or absence of bugs in the software modules to be predicted. However, no research has been conducted on bug prediction models built under the above realistic settings. Consequently, we established the following two objectives in this research. (1) We experimentally evaluate the prediction performance of bug prediction models built under realistic settings. (2) We propose techniques to improve the prediction performance of bug prediction models built under realistic settings. The first objective has now been achieved. Our experimental results show that the F-Measure of the bug prediction models built under realistic settings is only 0.19. Thus, there are still some issues to be solved to build a high-performance bug prediction model under realistic settings.