Using Single Demonstrations to Define Autonomous Manipulation Contact Tasks in Unstructured Environments via Object Affordances

Performing a manipulation contact task in an unknown and unstructured environment is still a challenge. Learning from Demonstration (LfD) techniques provide an intuitive means to define difficult-to-model contact tasks, but have attributes that make them undesirable for novice users in uncertain environments. We present a novel end-to-end system that captures a single manipulation task demonstration from an augmented reality (AR) head-mounted display (HMD), computes an affordance primitive (AP) representation of the task, and sends the task parameters to a mobile manipulator for execution in realtime. Using an AR HMD for task demon-stration and APs for task representation has several distinct advantages. AR task demonstration is intuitive, practical, and can be accomplished without requiring sensor installment in the task environment. APs provide a compact and legible task representation, enabling scalability, generalization, and modification of the task without significant data processing overhead. In this effort, we demonstrate system generalization with 10 object manipulation tasks, confirming the computed parameters from all tasks fit within AP tolerances. Secondly, we evaluate a mobile manipulator robot's ability to perform human-demonstrated tasks using AP representation. To increase robustness, we devised and tested four methods to correct for inherent, irreducible position errors in the system. A final study shows the system has a manipulation success rate of 96 % from a single manipulation demonstration on an industrial wheel valve.