Java Unit Testing Tool Competition - Fifth Round

After four successful JUnit tool competitions, we report on the achievements of a new Java Unit Testing Tool Competition. This 5th contest introduces statistical analyses in the benchmark infrastructure and has been validated with significance against the results of the previous 4th edition. Overall, the competition evaluates four automated JUnit testing tools taking as baseline human written test cases from real projects. The paper details the modifications performed to the methodology and provides full results of the competition.

I. Introduction

The key objective in the past four editions of the Java Unit Testing Tool Competition emerges from the need to advance research in the automated testing field. We have evidence from the results of the past edition [1]: each single automated testing tool performed worse than human written test cases. Using benchmarking [2] provides a means to mature the state of development of testing tools. Additionally, a benchmark can report useful data to interested parties, e.g. the software industry.