A Benchmarking Framework for Systematic Evaluation of Robotic Pick-and-Place Systems in an Industrial Grocery Setting

Robotic manipulation is a very active field of research nowadays; however, pick-and-place operations constitute the majority of today's industrial robotic applications. In order to adopt a robotic solution for an industrial setting, proper evaluation processes should be defined to assess the system's performance. A number of benchmarks have been proposed in the literature focusing mainly on individual components needed to perform the task, like grasping, perception and motion planning; thus, they do not provide enough information on the performance of the entire robotic system. To address this, we propose a benchmarking framework for a pick-and-place task inspired by a use case for picking fruits and vegetables in an industrial setting. To foster reproducible research and comparison of different robotic systems, the benchmarking framework uses surrogate objects with instructions on how to build them, an easy-to-reproduce environment, and guidelines for object placement. The proposed benchmark is applied to evaluate the performance of two variants of a robotic system with different end-effectors.

I. Introduction

Robotic manipulation is one of the most active areas in robotic research. Even though we have witnessed fascinating developments in the field, reproducibility and repeatability of studies are still challenging topics, rendering objective evaluation and building upon previous work a difficult or even impossible task [1].