A Method for Component Evaluation for Live Testing of Cloud Systems

Live testing is about testing a subsystem in production without causing any unacceptable disturbance to the production traffic. A subsystem is tested in production for multiple purposes such as deployment verification, fault prediction, fault localization, etc. The main challenge of live testing is alleviating the risk of test interferences as it may lead to a violation of a system's functional or non-functional requirements. To properly handle this risk, one needs to know which components present a risk of test interferences and what is the cost of the countermeasures to handle that risk. Existing literature relies heavily on human judgement, which can be time consuming, not always feasible, may provide misleading insight. In this paper we go through the challenges of automating this evaluation process and propose a solution to overcome them. Our solution consists of a method for components evaluation which goes through three steps, evaluation of test interferences that may manifest in external behaviour, evaluation of test interferences that may manifest in resource consumption, and finally the evaluation of the cost of implementing the countermeasures to overcome the risk of test interferences.