Advancing towards Safe Reinforcement Learning over Sparse Environments with Out-of-Distribution Observations: Detection and Adaptation Strategies

Safety in AI-based systems is among the highest research priorities, particularly when such systems are deployed in real-world scenarios subject to uncertainties and unpredictable inputs. Among them, the presence of long-tailed stimuli (Out-of-Distribution data, OoD) has captured much interest in recent times, giving rise to many proposals over the years to detect unfamiliar inputs to the model and adapt its knowledge accordingly. This work analyzes several OoD detection and adaptation strategies for Reinforcement Learning agents over environments with sparse reward signals. The sparsity of rewards and the impact of OoD objects on the state transition distribution learned by the agent are shown to be crucial when it comes to the design of effective knowledge transfer methods once OoD objects are detected. Furthermore, different approaches to detect OoD elements within the observation of the agent are also outlined, stressing on their benefits and potential downsides. Experiments with procedurally generated environments are performed to assess the performance of the considered OoD detection techniques, and to gauge the impact of the adaptation strategies on the generalization capability of the RL agent. The results pave the way towards further research around the provision of safety guarantees in sparse open-world Reinforcement Learning environments.

I. Introduction

As Artificial Intelligence (AI) based systems permeate various aspects of modern life, from healthcare to autonomous vehicles, ensuring their safe operation in deployment has become a paramount challenge. Safety in AI-based systems implies endowing them with the capability to identify and withstand hazards of different nature, from adversarial attacks, long tails and distribution shifts to systemic risks and inherent pitfalls of these systems, including machine ethics and their alignment with human goals and values [1]. Making AI-based systems robust against such hazards has engaged significant interest within the community in recent times [2], [3].