Grasping with Humanoid Hands based on In-Hand Vision and Hardware-accelerated CNNs

We present a vision-based grasping system for humanoid and prosthetic hands using hardware-accelerated CNNs for real-time object classification and class-aware pixel-wise segmentation. The system is implemented on the hand internal processing hardware of a humanoid hand using a System-on-Chip (SoC) comprising a Processor System (PS) and an FPGA. As a sensor system, the hand provides an integrated RGB camera, a multi-region Time-of-Flight (ToF) depth sensor, and an Inertial Measurement Unit (IMU). We propose an algorithm for 3D object shape estimation based on sensory information provided by the hand internal sensor system. The 3D object mesh in combination with the object relative pose of the hand is used as input for a reactive grasp controller. For the design of the CNN-based object recognition and segmentation networks, we use a resource-aware algorithm for Network-Architecture-Synthesis (NAS). We evaluate the visual perception accuracy and 3D model estimation accuracy in grasping experiments with six objects. We obtain a mean object segmentation accuracy of 84.4 % and a mean error for object diameter estimation of 44 mm.