High-Resolution 3-Dimensional Contact Deformation Tracking for FingerVision Sensor With Dense Random Color Pattern

Recent studies on vision-based tactile sensing have shown promising results on the perception of contact information, which could improve the performance of dexterous manipulations. However, 3-dimensional contact deformation tracking at a higher resolution is desired and remains a challenge for vision-based tactile sensors with monocular configurations. In this work, a similar hardware structure to our previous FingerVision sensor is adopted. The dot markers are replaced with a novel random color pattern as the sensor's tracking target and a dense optical flow algorithm is used to track the deformation of its elastic contact interface. This results in a more accurate 2-dimensional deformation field estimation at a higher resolution in comparison with that obtained using sparse dot markers. Additionally, the denser and more accurate deformation field enables depth estimation with better fidelity. To achieve depth estimation purely from the optical flow field, Gaussian density feature extraction and processing framework are proposed. The resulting depth map can be used independently as a tactile sensing modality, or jointly with the accurate in-plane displacement field as a more complete deformation tracking of contact interfaces for vision-based tactile sensors.