Benchmarking In-Hand Manipulation

The purpose of this benchmark is to evaluate the planning and control aspects of robotic in-hand manipulation systems. The goal is to assess the system's ability to change the pose of a hand-held object by either using the fingers, environment or a combination of both. Given an object surface mesh from the YCB data-set, we provide examples of initial and goal states (i.e. static object poses and fingertip locations) for various in-hand manipulation tasks. We further propose metrics that measure the error in reaching the goal state from a specific initial state, which, when aggregated across all tasks, also serves as a measure of the system's in-hand manipulation capability. We provide supporting software, task examples, and evaluation results associated with the benchmark.

I. Introduction

In-hand manipulation is the task of changing the grasp on an hand-held object without placing it back and picking it up again. In recent years, researchers have demonstrated in-hand manipulation skills on real robotic platforms with end-effectors ranging from simple grippers [1] to anthropomorphic hands [2]–[5] on everyday objects (some examples shown in Fig. 1). This growing interest in in-hand manipulation has created the need for a thorough comparison of the proposed methods using a common set of tasks and metrics. However, the diversity in the methodological approaches and hardware used makes the burden of comparison rather complex for individual research labs to perform. There is thus a need for a benchmarking protocol that allows researchers to evaluate new methodologies in a more standardized manner.