PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles

Even though virtual testing of Autonomous Vehicles (AVs) has been well recognized as essential for safety assessment, AV simulators are still undergoing active development. One particular challenge is the problem of including the Sensing and Perception (S&P) subsystem into the virtual simulation loop in an efficient and effective manner. In this article, we define Perception Error Models (PEM), a virtual simulation component that can enable the analysis of the impact of perception errors on AV safety, without the need to model the sensors themselves. We propose a generalized data-driven procedure towards parametric modeling and evaluate it using Apollo, an open-source driving software, and nuScenes, a public AV dataset. Additionally, we implement PEMs in SVL, an open-source vehicle simulator. Furthermore, we demonstrate the usefulness of PEM-based virtual tests, by evaluating camera, LiDAR, and camera-LiDAR setups. Our virtual tests highlight limitations in the current evaluation metrics, and the proposed approach can help study the impact of perception errors on AV safety.

I. Introduction

One of the major limitations preventing a public deployment of Autonomous Vehicles (AVs) is that their safety is not yet guaranteed or well-established in general. Furthermore, Shen et al. [1] emphasize how the explainability of Artificial Intelligence (AI) in AVs is necessary for trust and acceptance. Hence, the automotive industry, the academic community, and regulators are developing safety assessment procedures. By analyzing the AV Operational Design Domain (ODD) [2], we can identify the challenges that the AV will encounter. In particular, sensors are known to be very susceptible to weather conditions and the amount of sunlight [3]. The AV should handle such issues and still exhibit safe driving behavior. For example, if the perception uncertainty increases, the AV could reduce its speed and adopt a more defensive driving, thus maintaining an adequate level of safety. Nevertheless, failures in obstacle detection may still lead to undesirable behavior such as collisions, emergency maneuvers, or traffic rules violations. For instance, the leading cause of a 2018 AV fatal accident was determined to be a perception error that was not adequately handled [4]. Thus, a deeper understanding of how perception errors affect the AV response is necessary for safety assurance.