Vision-Based Dexterous Motion Planning by Dynamic Movement Primitives with Human Hand Demonstration

This paper proposes a vision-based framework for a 7 -degree-of-freedom robotic manipulator, with the primary objective of facilitating its capacity to acquire information from human hand demonstrations for the execution of dexterous pick-and-place tasks. Most existing works only focus on the position demonstration without considering the orientations. In this paper, by employing a single depth camera, MediaPipe is applied to generate the three-dimensional real-world coordinates of a human hand, thereby comprehensively recording the hand’s motion, encompassing the trajectory of the wrist, orientation of the hand, and grasp motion. A mean filter is applied during data pre-processing to smooth the raw data. The demonstration is designed to pick up an object at a specific angle, navigate around obstacles in its path, and subsequently, deposit it within a sloped container. The robotic system demonstrates its learning capabilities, facilitated by the implementation of dynamic movement primitives, enabling the assimilation of user actions into its trajectories with different start and end points. Experimental studies are carried out to demonstrate the effectiveness of the work.