A Virtual Hand Based Manipulation Learning Framework for Dexterous Hand

Dexterous hand programming is considered to be generally difficult suffering from high degree of freedom. Vision-based manipulation learning provides an efficient way to support automatic programming by human demonstration. However, complex visual perception is tightly coupled with the kinematics structure of dexterous hand in existing solutions, which makes it hard to migrate the developed algorithm from one kind of dexterous hand to another due to their kinematic difference. To address the issue, we introduce an intermediary called virtual hand and a new paradigm of motion mapping, serving as the link between human hand and dexterous hand. The virtual hand is generated in real time by hand pose estimation to track the motion of human hand. Then the joint configurations of dexterous hand and robotic arm can be calculated using geometry and optimization method based on the motion mapping paradigm we proposed. The method we introduced decouples the visual perception and kinematics calculation, which brings flexibility and low cost. The results of our preliminary experiments show the effectiveness of our method.

I. Introduction

Dexterous hands are human-like tools designed to imitate the agility and strength of human hands. It is known that programming dexterous hands can be difficult and tedious due to their high degree of freedom (DOF). Manipulation learning offers a convenient solution to automatically program dexterous hands by releasing programmers from the burden of complicated kinematics calculations. One way to accomplish this has been using external sensors, like joysticks [1] and wearable devices [2]–[4], to capture data on joint or keypoint motions of human hands, and then translating that information onto the dexterous hand itself. Despite its effectiveness, these traditional methods require additional hardware/devices to gather information, yet practicality raises questions over price and ease of use.