Evaluation Uncertainty in Data-Driven Self-Driving Testing

Safety evaluation of self-driving technologies has been extensively studied. One recent approach uses Monte Carlo based evaluation to estimate the occurrence probabilities of safety-critical events as safety measures. These Monte Carlo samples are generated from stochastic input models constructed based on real-world data. In this paper, we propose an approach to assess the impact on the probability estimates from the evaluation procedures due to the estimation error caused by data variability. Our proposed method merges the classical bootstrap method for estimating input uncertainty with a likelihood ratio based scheme to reuse experiment outputs. This approach is economical and efficient in terms of implementation costs in assessing input uncertainty for the evaluation of self-driving technology. We use an example in autonomous vehicle (AV) safety evaluation to demonstrate the proposed approach as a diagnostic tool for the quality of the fitted input model.

I. Introduction

The competitive race toward the mass deployment of self-driving cars driving side-by-side with human-driven vehicles on public roads advocates for an accurate and highly precise safety evaluation framework to ensure safe driving. However, achieving meaningful precision is a challenging task when the safety-critical events under study are rare in naturalistic situations. A recent method has been developed which which adopts Monte Carlo method empowered by the Importance Sampling technique as a variance reduction scheme; this has produced appealing results. In [1], it is shown that the efficiency is enhanced by ten thousand times with the incorporation of large-scale driving data sets and the employed statistical models. This improved efficiency is highly appealing for autonomous vehicle (AV) researchers as the required testing effort is overly demanding, an estimate of 8.8 billion driving miles required to provide ‘sufficient’ evidence to compare the safety of AV driving and human driving from logged data [2].