Experimentation in the Operating System: The Windows Experimentation Platform

Online controlled experiments are the gold standard for evaluating improvements and accelerating innovations in online and app worlds. However, little is known about applicability, implementation, and efficacy of experimentation for operating systems (OS), where many features are non-user-facing. In this paper, we present the Windows Experimentation platform (WExp), and insights from implementation and execution of real-world experiments in the OS. We start by discussing the need for experimentation in OS, using real experiments to illustrate the benefits. We then describe the architecture of WExp, focusing on unique considerations in its engineering. Finally, we discuss learnings and challenges from conducting real-world experiments. Our experiences and insights can motivate practitioners to start experimenting as well as to help them to successfully build their experimentation platforms. The learnings can also guide experimenters with best-practices and highlight promising avenues for future research.

I. Introduction

Online controlled experiments, aka A/B tests, are used by many organizations, e.g. Google, Facebook, Amazon, and Microsoft, to design and improve their apps and online services [1]–[4]. However, little is known about experimentation in the operating system, where features being experimented with are often not user facing.