A Smart Interactive Camera Robot Based on Large Language Models

The emergence of large language models (LLMs) has paved the way for advancing robotics capabilities, especially in intricate tasks that demand nuanced comprehension and precision. In this context, this paper introduces a novel interactive camera robot that harnesses LLMs to enhance human-robot interaction and optimize robot control. Specifically, an innovative technique that leverages the language understanding capabilities of LLMs to plan camera movement trajectories and waypoints is presented. In this study, the geometric relationships among the objects under capture are employed to plan the control strategy. Accordingly, this approach not only empowers sophisticated camera parameter manipulation and color adjustments but also fosters a natural and efficient human-robot interaction. Lots of experiments on real robots are conducted to evaluate the effectiveness of the proposed method under various scenarios. The results reveal robust performance across crucial measures, affirming the substantial potential of LLMs in elevating camera robot control and interaction experience. Videos of our experiments are available at https://youtu.be/zP-sTZHvXe4.

I. Introduction

Large language models (LLMs) have emerged as a promising avenue for acquiring a comprehensive understanding of the world by transforming it into a linguistic framework. Nonetheless, utilizing such broad knowledge to enable embodied agents to execute precise, specific physical actions in the real world remains a challenge. To handle such issues, some studies encapsulate the agent’s actions into action primitives to promote LLM invocation [1], [2]. Conversely, others integrate reinforcement learning with LLMs [3], [4], or even let LLMs define rewards to guide agents to accomplish intricate implementations of fundamental actions, such as grasping and navigation tasks [5]–[7].