Imitating Tool-Based Garment Folding From a Single Visual Observation Using Hand-Object Graph Dynamics

Garment folding is a ubiquitous domestic task that is difficult to automate due to the highly deformable nature of fabrics. In this article, we propose a novel method of learning from demonstrations that enables robots to autonomously manipulate an assistive tool to fold garments. In contrast to traditional methods (that rely on low-level pixel features), our proposed solution uses a dense visual descriptor to encode the demonstration into a high-level hand-object graph (HoG) that allows to efficiently represent the interactions between the manipulated tool and robots. With that, we leverage graph neural network to autonomously learn the forward dynamics model from HoGs, then, given only a single demonstration, the imitation policy is optimized with a model predictive controller to accomplish the folding task. To validate the proposed approach, we conducted a detailed experimental study on a robotic platform instrumented with vision sensors and a custom-made end-effector that interacts with the folding board.

I. Introduction

Robots have been extensively used to support people in a variety of activities of daily living. Garment folding is a clear example of a monotonous service task that can theoretically be performed by robots but which, in practice, is difficult to solve by using these state-of-the-art strategies [1], [2]. One possible solution to alleviate the complexity of manipulating fabrics is to enable the robot to learn how to use an assistive tool by observing an expert demonstration and then imitating the behavior. This approach is typically referred to as imitation learning (IL) [3], [4], a technique that enables autonomous agents (e.g., robots) to acquire complex skills from simple sensory data without requiring to hard-code the strategies. Our aim in this work is to solve the garment folding problem by using an assistive tool under the IL paradigm.