Reinforcement learning with incremental skill models: Extension to tool use as skills

Considering manipulation of various tools and objects, such as the cooking task of robots, various methods of acquiring behaviors have been proposed, but reusing the behavioral knowledge acquired by robots and acquiring generic knowledge remain difficult issues. In our model-based reinforcement learning using the skill library, the behavior of the robot is represented as a graph of skills, and the dynamics of each skill are modeled for motion planning. This method has the advantage that the learned dynamics model can be reused in other tasks and situations, and we believe that this will be the basis for a general-purpose robot motion planning method. In model-based reinforcement learning using the skill library, the introduction of tools into the existing skill library has not been considered. Adding tool-use skills into the skill library, we think resolvable situations will expand more than ever. For example, if we introduce a funnel into the liquid pouring task, we can change the dynamics of the liquid flow to make it less spillage. On the other hand, from the viewpoint of dynamics model design, modeling the flow dynamics in a way that is not affected by the use of funnels is preferable in terms of reusability. In this study, we introduce Squeeze skill (squeeze out with a soft container) and Mount Funnel skill (enlarge the receiving mouth with a funnel) into the liquid pouring task as a case study of motion planning using model-based reinforcement learning using the skill library introduced tool skills. We design dynamics models that do not depend on the addition of tools and show that the robot acquires motions that appropriately use tools according to the situation. We also analyze the effect of learning a tool-use skill dynamics model in combination with other trained dynamics models.