The complexities of grasping in the wild

The recent ubiquity of high-framerate (120 fps and higher) handheld cameras creates the opportunity to study human grasping at a greater level of detail than normal speed cameras allow. We first collected 91 slow-motion interactions with objects in a convenience store setting. We then annotated the actions through the lenses of various existing manipulation taxonomies. We found manipulation, particularly the process of forming a grasp, is complicated and proceeds quickly. Our dataset shows that there are many ways that people deal with clutter in order to form a strong grasp of an object. It also reveals several errors and how people recover from them. Though annotating motions in detail is time-consuming, the annotation systems we used nevertheless leave out important aspects of understanding manipulation actions, such as how the environment is functioning as a “finger” of sorts, how different parts of the hand can be involved in different grasping tasks, and high-level intent.

I. Introduction

For roboticists working on dexterous robots, observation of human manipulation continues to be an important way to understand the problem of grasping (e.g. [1]–[3]). One way of observing human grasping in detail is to use high-framerate video. Due to the growing ubiquity of high-framerate video cameras in phones, it is now feasible to capture a large number of grasping actions “in the wild” i.e. in everyday settings such as cluttered workspaces. The large number of actions and the everyday setting allows behaviors such as mistakes to be captured, and the high framerate reveals detailed finger movement and the making and breaking of contact.