PTFD-Net: A Sliding Detection Algorithm Combining Point Cloud Sequences and Tactile Sequences Information

Sliding detection can effectively enhance the stability of robot grasping operations. Methods relying solely on 2-D vision or tactile information for sliding detection often exhibit limited applicability across diverse scenarios. Furthermore, 2-D visual sensitivity to the changes in light intensity leads to inadequate performance in overexposure scenarios. In this article, we propose a novel deep neural network called point cloud sequences and tactile sequences fusion detection network (PTFD-Net), which serves the purpose of detecting the sliding state when robots grasp objects. Specifically, we utilize an improved PointRNN algorithm and 3-D convolution to extract features from the point cloud sequences and tactile sequences to construct a multimodal sequence information prediction model. In addition, we construct a dataset that integrates point cloud sequences with tactile sequences. The impact of different point cloud sequence lengths and tactile image sizes on the performance of PTFD-Net is discussed to determine the optimal parameters. The model can attain a sliding state detection accuracy of 95.49% on the dataset. Moreover, we conduct performance comparisons with baseline algorithms under overexposure scenarios, demonstrating PTFD-Net’s superiority over baseline algorithms and achieving a sliding detection accuracy of 92.71%. Finally, we conduct robustness experiments on unknown objects under different grasping poses and occlusion scenarios, resulting in sliding detection accuracies of 86.02% and 89.43%, respectively, both surpassing single-modal models. These results demonstrate the effectiveness of PTFD-Net in extracting multimodal features, enabling it to perform better robot grasping tasks in complex environments.