Reinforcement Learning Control With Knowledge Shaping

We aim at creating a transfer reinforcement learning framework that allows the design of learning controllers to leverage prior knowledge extracted from previously learned tasks and previous data to improve the learning performance of new tasks. Toward this goal, we formalize knowledge transfer by expressing knowledge in the value function in our problem construct, which is referred to as reinforcement learning with knowledge shaping (RL-KS). Unlike most transfer learning studies that are empirical in nature, our results include not only simulation verifications but also an analysis of algorithm convergence and solution optimality. Also different from the well-established potential-based reward shaping methods which are built on proofs of policy invariance, our RL-KS approach allows us to advance toward a new theoretical result on positive knowledge transfer. Furthermore, our contributions include two principled ways that cover a range of realization schemes to represent prior knowledge in RL-KS. We provide extensive and systematic evaluations of the proposed RL-KS method. The evaluation environments not only include classical RL benchmark problems but also include a challenging task of real-time control of a robotic lower limb with a human user in the loop.

I. Introduction

Transfer learning in the context of reinforcement learning is less studied than that in supervised or unsupervised learning [1], [2], [3], [4]. In this study, we investigate knowledge transfer in RL by virtual task. We consider a task to be described by the components of an RL problem, i.e., the state space, the action space, the system dynamics or the environment, and the rewards or costs. A successful knowledge transfer is expected to improve learning performance if knowledge from source tasks can be efficiently utilized in learning a new target task.