Using benchmarking to advance research: a challenge to software engineering

Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.

1. Introduction

The purpose of this paper is to issue a challenge to software engineering to start defining benchmarks for the key problems in their area. From our experience developing benchmarks and examining other successful benchmarks, we have observed that benchmarking, when embraced by a community, has a strong positive effect on the scientific maturity of a discipline.