Testing a non-deterministic robot in simulation - How many repeated runs ?

Testing autonomous robots requires test campaigns in the field that could be expensive, risky for the system and its environment, or even impossible to set up. To cope with these limits, an expanding approach is to perform tests in virtual environments using simulators. Due to the non-determinism of the robot control and simulation execution, a test might fail or pass with exactly the same inputs (same world and mission). An important question is thus: how many runs are required to activate a fault? We particularly focus in this paper on the efficiency of repeated runs and world diversity in the context of functional testing. We address this research question with an industrial case study - an agricultural weeding robot developed by Naïo Technologies. We conclude that, in this case study, test input diversity is more efficient than repeated runs in order to activate faults. More importantly, we propose an experimental approach to assess the impact of the non-determinism, which may be reused in other case studies.