Benchmarking Protocol for Grasp Planning Algorithms

Numerous grasp planning algorithms have been proposed since the 1980s. The grasping literature has expanded rapidly in recent years, building on greatly improved vision systems and computing power. Methods have been proposed to plan stable grasps on known objects (exact 3D model is available), familiar objects (e.g. exploiting a-priori known grasps for different objects of the same category), or novel object shapes observed during task execution. Few of these methods have ever been compared in a systematic way, and objective performance evaluation of such complex systems remains problematic. Difficulties and confounding factors include different assumptions and amounts of a-priori knowledge in different algorithms; different robots, hands, vision systems and setups in different labs; and different choices or application needs for grasped objects. Also, grasp planning can use different grasp quality metrics (including empirical or theoretical stability measures) or other criteria, e.g., computational speed, or combination of grasps with reachability considerations. While acknowledging and discussing the outstanding difficulties surrounding this complex topic, we propose a methodology for reproducible experiments to compare the performance of a variety of grasp planning algorithms. Our protocol attempts to improve the objectivity with which different grasp planners are compared by minimizing the influence of key components in the grasping pipeline, e.g., vision and pose estimation. The protocol is demonstrated by evaluating two different grasp planners: a state-of-the-art model-free planner and a popular open-source model-based planner. We show results from real-robot experiments with a 7-DoF arm and 2-finger hand, as well as simulation-based evaluations.

I. Introduction

Grasp planning has been studied since the 1980s [1], with a recent proliferation of novel approaches. Different methods assume different a-priori knowledge, e.g. model-based [2] versus model-free [3], and adopt widely differing approaches, e.g. analytic [4], [5], data-driven [6], [7], or based on human demonstration [8]. Other work explores mechanically adaptive hands that simplify the grasping process thanks to their inherent mechanical adaptability [9], [10], or combines different sensing modalities for performing grasping [11] and in-hand manipulation [12]. Recent reviews [13]–[16] categorise and discuss these algorithms in terms of their differences, assumptions and limitations. Competitions such as the Amazon Picking Challenge (APC) [17] and Robotic Grasping and Manipulation Competition at IROS [18] have proposed different tasks to compare the performance of whole robotic systems. While stimulating significant progress, such contests can also engender over-fitting of engineering solutions to the proposed tasks.