Long-Horizon Planning and Execution With Functional Object-Oriented Networks

Following work on joint object-action representations, functional object-oriented networks (FOON) were introduced as a knowledge graph representation for robots. A FOON contains symbolic concepts useful to a robot's understanding of tasks and its environment for object-level planning. Prior to this work, little has been done to show how plans acquired from FOON can be executed by a robot, as the concepts in a FOON are too abstract for execution. We thereby introduce the idea of exploiting object-level knowledge as a FOON for task planning and execution. Our approach automatically transforms FOON into PDDL and leverages off-the-shelf planners, action contexts, and robot skills in a hierarchical planning pipeline to generate executable task plans. We demonstrate our entire approach on long-horizon tasks in CoppeliaSim and show how learned action contexts can be extended to never-before-seen scenarios.

I. Introduction

An Ongoing trend in robotics research is the development of robots that can jointly understand human intention and action and execute manipulations for human domains. A key component for such robots is a knowledge representation that allows a robot to understand its actions in a way that mirrors how humans communicate about action [1]. Inspired by the theory of affordance [2] and prior work on joint object-action representation [3], the functional object-oriented network (FOON) was introduced as a knowledge graph representation for service robots [4], [5]. FOONs describe object-oriented manipulation actions through its nodes and edges and aims to be a high-level planning abstraction closer to human language and understanding. They can be automatically created from video demonstrations [6], and a set of FOONs can be merged into a single network from which knowledge can be quickly retrieved as plan sequences called task trees [4].