Effects of an Economic Approach for Test Case Selection and Reduction for a Large Industrial Project

Extensive testing in large projects can lead to tremendous test costs with superlinear growth over time. Researchers have proposed several techniques to tackle this problem. However, the practical effects of these techniques on the asymptotic behaviour of test costs growth in large industrial software projects remains poorly characterized. We introduce and analyse a fixed time budget for test executions for SAP HANA, a large industrial project. Our approach assigns a global fixed time budget to several components. Each component can only execute tests within its budget, which can change only by transfers from other components. This limits the number of test executions for each test run to a constant, thus reducing the asymptotic growth of test costs. Budget transfers and test optimizations adhere to balances between value and costs, thus creating an economic environment for test case selection and reduction. Specifically, this creates incentives to remove unnecessary tests and to optimize test execution times. For SAP HANA, our approach leads to effective test case selection and reduction, and reduces test execution times by 105 years in four months with a negligible effect on quality. The trade-off between runtime savings and failure detection is 1.83 years per failure.

I. Introduction

Software testing is a fundamental part of quality assurance for software projects. Agile development processes, continuous integration and short release cycles require frequent automatic test runs to reduce the number of defects and to prevent regressions during development.