A Vision-Based Tactile Sensing System for Multimodal Contact Information Perception via Neural Network

Typically, robotic dexterous hands are equipped with various sensors to acquire multimodal tactile information, which is an important way for robots to perceive and interact with the environment. Vision-based tactile sensors have been widely developed due to their simple structure and high resolution. It adopts a specialized optical design to convert different contact information into different signal responses, such as recognizing contact force information based on the marker-based method, obtaining contact shape information based on the photometric stereo (PS) method, and so on. However, the current mainstream vision-based tactile sensing systems tend to adopt isolated optical design strategies and different data processing methods for different modal tactile information, introducing limitations in system integration. This article proposes a vision-based tactile sensing strategy for multimodal tactile information using only reflected light field information. The specific implementation of the vision-based tactile sensor and the recognition algorithm for sensing multiple tactile information simultaneously are described in detail. The results show that the tactile sensing strategy does not need to design different optical structures for different modal tactile information but only uses a simple reflection layer combined with a neural network to sense multimodal tactile information, which reduces the complexity of the tactile system. In addition, the system achieves a force error of 0.2 N and a pose error of 0.41° and shows excellent precision and recall in localization and classification tasks, demonstrating the potential for multimodal tactile integration in various fields.