Intent-based Object Grasping by a Robot using Deep Learning

A robot needs to predict an ideal rectangle for optimal object grasping based on the intent for that grasp. Mask Regional - Convolutional Neural Network (Detectron) can be used to generate the object mask and for object detection and a Convolutional Neural Network (CNN) can be used for ideal grasp rectangle prediction according to the supplied intent, as described in this paper. The masked image obtained from Detectron along with the metadata of the intent type has been fed to the Fully-Connected layers of the CNN which would generate the desired optimal rectangle for the specific intent and object. Before settling for a CNN for optimal rectangle prediction, conventional Neural Networks with different hidden layers have been tried and the accuracy achieved was low. A CNN has then been developed and tested with different layers and sizes of pool and strides to settle on the final CNN model that has been discussed here. The optimal predicted rectangle is then fed to a robot, ROS simulation of Baxter robot in this case, to perform the actual grasping of the object at the predicted location.