HMI Design for Explainable Machine Learning Enhanced Risk Detection in Low-Altitude UAV Operations

Unmanned aerial vehicles (UAVs) are increasingly utilized in low-altitude urban airspace for various applications. However, operating UAVs in populated areas poses safety risks, including proximity to people and property, unauthorized flight, and collisions with obstacles or hazards such as wildfires. Safety systems like geo-fencing and collision avoidance systems can detect known, well-defined risks but fail to address real-time detection of unforeseen hazards. To tackle this challenge, the paper proposes using an onboard camera on the UAV to continuously scan the environment. The captured video is analyzed in real-time using machine learning (ML) to detect, locate and classify a limited set of common hazards, determine their severity, and reroute the UAV to ensure flight safety. In situations where autonomous rerouting is deemed too risky by the system, the remote pilot is alerted, provided with detailed information on the detected hazards including the camera imagery, and is thus enabled to make an informed decision. Here, understanding the ML-model's results and decisions is crucial for effective decision-making, yet ML-models are often opaque and difficult for humans to comprehend. To address this issue, the paper suggests integrating principles of explainable artificial intelligence (XAI) into the human-machine interface (HMI) design to enhance interpretability and make the ML-models transparent and comprehensible. Specifically, principles of Class Activation Mapping (CAM) and Layerwise Relevance Propagation (LRP) are combined with a hierarchical HMI to present the information to the remote pilot. The paper describes relevant XAI-principles for HMI design and develops methods to comprehensively display ML-based information to remote pilots. These display design principles are then applied and integrated into an HMI design framework for supervising multiple UAVs. The resulting XAI-display prototype is discussed, limitations are identified, and further research steps are outlined for implementing the ML-models and testing the HMI with subject matter experts.