Integrating Computer Vision and Natural Language Instruction for Collaborative Robot Human-Robot Interaction

In this paper, we report how we design a human-robot co-working scheme. We believe that natural language is the best way that human and machine can exchange messages. However, understand voice command is not enough, what object is referred in what location is also very import. To better understand a natural language instruction, a robot also require computer vision to resolve the real-world object referencing problem. In this paper we show how to integrate the Google voice recognition result with the YOLO computer vision software with the KINOVA robot arm and achieve a collaborative robot interaction.