Towards estimation of human intent in assistive robotic teleoperation using kinaesthetic and visual feedback

The ability to predict human intent in manipulating in-hand objects is a crucial aspect of developing intelligent robotic systems that can effectively interact with and assist humans in various tasks. Due to the non-standardized nature of interfaces between different robots, it is non-trivial to establish a one-to-one mapping between the instructions provided by the human operator on to the robot, and vice-versa. Additionally, the round trip of information flow in move-and-wait teleoperation strategy for micro-instructions accumulates considerable delays in performing basic tasks, rendering the overall objective ineffective. In this context, predicting the human intent of teleoperation is a prospective strategy to mitigate the effect of such delays. This entails that a possible set of expected action(s) is to be represented ahead of time. In this study, we propose an ML-driven ensemble approach for estimating the goal pose configuration of an object of interest held within the end-effectors of a remotely connected robot using visual and kinematic measurements. We evaluate our proposed system to infer the intended action of a human operator in a real-world robotic setup involving a haptic glove and a dexterous robotic hand, on three different objects. The proposed methodology outperforms a benchmark model in literature utilizing 60 times lesser prediction time with substantially better performance. We provide a comparative analysis of intent prediction strategy using independent visual and kinaesthetic data and discuss its improvement when combining both the modalities.