Virtual Reality Based Tactile Sensing Enhancements for Bilateral Teleoperation System With In-Hand Manipulation

Tactile sensing is important for contact-rich tasks especially in where an in-hand manipulation is involved. In teleoperation, such feedback can provide information of when and where the contacts happen, and is essential for a human operator to make appropriate actions. To improve the experience in human-robot interaction in teleoperation without vision feedback, in this letter, a model-based sensing enhancement system is proposed. This system allows a human operator to remotely control a dexterous robotics hand with a contact feedback in the form of haptics. Under this framework, we introduce a calibration method to map the hand joint movements of the master and the slave. Given noisy robot sensory data, a learning based approach is adopted to estimate the object pose online. The estimated pose is sent to Unity, of which is leveraged to calculate the hand-object contact force. Finally, this force is rendered to the master, a wearable haptic device, worn by the operator. Our experiments have shown that with this contact information, the performance surpassed those conducted on a bilateral teleoperation system without sense of contact. Additionally, the user can safely manipulate the object with the robot’s hand.