Action2Code: Transforming Video Demonstrations into Sequential Robotic Instructions

The time-consuming nature of programming robotic motion and manipulation to achieve specific tasks is a key reason that robots have generally been restricted to repetitive tasks with little variation. Developers need to manually write specific code for each task, making it challenging to adapt the code to different environments and assembly scenarios. This inefficiency leads to significant time being spent on creating redundant code for similar robotic actions in various assemblies. To address this, we develop a generative network to program robots through demonstration, bringing more agility to the process. We propose a network, Action2Code, that takes video demonstration as an input and translates it to robotic instructions. We evaluate our approach on two datasets, namely RH20T and ASMCode, using quantitative analysis across four metrics to assess code generation. We observe a CodeBLEU score of 0.69 and 0.65, along with a CodeBERTScore of 0.78 and 0.73 for RH20T and ASMCode, respectively. Our results demonstrate the effectiveness of the model in generating code from video demonstrations.