A Gesture- and Speech-Guided Robot Teleoperation Method Based on Mobile Interaction With Unrestricted Force Feedback

Operational devices used in the traditional robot teleoperation methods restrict the space for interaction. Furthermore, operational and feedback devices are often separated. Thus, the operator has to switch his attention between the operation area and the feedback area during the interaction, which is unnatural and inefficient. In this article, we propose a new gesture- and speech-guided teleoperation method, which unifies operation, visual feedback, and force feedback areas, to achieve a more immersive and natural interaction environment. First, a mobile interaction method combining mobile operation platform and augmented reality (AR) technology is proposed to enlarge the operational space of operators. Second, the gesture tracking method with AR allows the operators to directly interact with virtual objects using bare hands for natural interaction. Third, speech interaction is introduced to improve the precision of fine teleoperations and the naturalness of interactions. Fourth, we introduce unrestricted force feedback based on electromagnetic force technology to provide operators with force feedbacks without limiting their actions. Besides, a virtual guiding force based on the artificial potential field is presented to help the operators to complete high-precision tasks more efficiently. This approach allows the operators to focus on the task without switching attention and the experimental results show that the proposed method provides an efficient and natural interaction for robot teleoperation.