What works for whom, where, when, and why? On the role of context in empirical software engineering

Context is a central concept in empirical software engineering. It is one of the distinctive features of the discipline and it is an indispensable part of software practice. It is likely responsible for one of the most challenging methodological and theoretical problems: study-to-study variation in research findings. Still, empirical software engineering research is mostly concerned with attempts to identify universal relationships that are independent of how work settings and other contexts interact with the processes important to software practice. The aim of this paper is to provide an overview of how context affects empirical research and how empirical software engineering research can be better ‘contextualized’ in order to provide a better understanding of what works for whom, where, when, and why. We exemplify the importance of context with examples from recent systematic reviews and offer recommendations on the way forward.

1. INTRODUCTION

What is best? Pair programming or solo programming? Test-first or test-last? A multitude of studies have been performed to answer these and other similar questions of the type: “Is <technology x> better than <technology y?” However, asking the general question of whether pairs outperform individuals in programming tasks, or whether test-driven development results in higher productivity, is meaningless. It is meaningless since these questions can be