Enhancing Robot Perception in Grasping and Dexterous Manipulation through Crowdsourcing and Gamification

Robot grasping and manipulation planning in unstructured and dynamic environments is heavily dependent on the attributes of manipulated objects. Although deep learning approaches have delivered exceptional performance in robot perception, human perception and reasoning are still superior in processing novel object classes. Moreover, training such models requires large datasets that are generally expensive to obtain. This work combines crowdsourcing and gamification to leverage human intelligence, enhancing the object recognition and attribute estimation aspects of robot perception. The framework employs an attribute matching system that encodes visual information into an online puzzle game, utilizing the collective intelligence of players to expand an initial attribute database and react to real-time perception conflicts. The framework is deployed and evaluated in a proof-of-concept application for enhancing object recognition in autonomous robot grasping and a model for estimating the response time is proposed. The obtained results demonstrate that given enough players, the framework can offer near real-time labeling of novel objects, based purely on visual information and human experience.