Real time robotic arm control using hand gestures with multiple end effectors

This article describes experiments that explore the possibility of using an optical tracking device input to remotely control dual-arm robots. We propose using the Leap Motion controller as an alternative to using joysticks, this allows for more intuitive 6-DOF control where only one hand is needed to control each robot arm. We affixed two end effectors to a Baxter research robot, a standard electric gripper, and an AR10 robotic hand. The standard electric gripper was controlled via pinch gestures and all five fingers and thumb were controlled on the AR10 robotic hand using finger movement from the Leap. Some simple tests were performed and the results indicate that in lifting standard objects the standard electric gripper had a higher success rate. The main problem with the hand appeared to be due to the absence of any touch or force feedback to the operator, as the user became more comfortable with the system the better their performance became. This leads us to believe that the use of a robotic hand could be improved with a training program.