Shake Those System Parameters! On the Need for Parameter Coverage for Decision Systems

Decision systems such as Multiple-Criteria Decision Analysis systems formulate a decision process in terms of a mathematical function that takes into consideration different aspects of a problem. Testing such systems is crucial, as they are usually employed in safety-critical systems. A good test suite for these systems should be able to exercise all the possible types of decisions that can be taken by the system. Classic structural coverage criteria do not provide good test suites in this sense, as they can be fulfilled by simple tests that only cover one possible type of decision. Thus, in this paper we discuss the need for tailored coverage criteria for this class of systems, and we propose a criterion based on the perturbation of the decision systems’ parameters. We demonstrate the effectiveness of the criterion, compared to classic structural coverage criteria, on a path planner system for autonomous driving. We also discuss other benefits, such as the criterion helping explain why a decision was made during a test.