Intelligent Soft Robotic Gripper Enabled by Multimodal Sensors and Deep Learning

Here we report an intelligent soft robotic gripper enabled by the integration of an ultrasonic remote sensor and triboelectric sensors. Due to the noncontact distance sensing ability, the ultrasonic sensor is used to find the object’s visual information including position and height by lateral scanning. The information is then used for adjusting the robotic gripper to an appropriate grasp location, after which grasp operation is performed to obtain the object’s tactile information through triboelectric bending and tactile sensors. To efficiently analyze the multimodal information, a deep-learning neural network based on feature-level data fusion is constructed, which is able to achieve a high accuracy of 99.3% in classifying 14 objects, enabling the intelligent soft robotic gripper for various smart applications.