Policy Transfer in POMDP Models for Safety-Critical Autonomous Vehicles Applications

Developing models of complex systems, such as Autonomous Vehicles (AVs) that operate in uncertain, reactive environments is challenging because of complexity and uncertainty in their operational environments. Partially Observable Markov Decision Process (POMDP) is a mathematically principled framework that is suitable for developing such models. A POMDP model is typically defined with respect to a specific task performed in the environment and the resulting policy is obtained based on the defined task and information available about the environment. This implies that if changes occur in the task and/or environment, the model and policy need to be updated. One efficient way to achieve this end is to reuse the experience from the initial task(s) to perform similar task(s) instead of designing new models from scratch. This technique is referred to as Transfer Learning (TL). In this paper, we propose a novel TL technique that uses a customized Q-learning algorithm for policy transfer in POMDPs. We use this technique to transfer policies from a POMDP model, initially designed for a simple AV lane-keeping task within a freeway environment, to a similar task in a more complex environment, where random and risky behaviors from neighboring vehicles in the environment can be expected.