Dynamic Task Planning: An Integrated Approach with Scene Relation Perception and Knowledge Graphs

Confronted with the limitations of conventional Planning Domain Definition Language (PDDL) in dynamic and unpredictable kitchen environments, this paper introduces a novel task planning methodology that leverages the synergy between advanced scene graphs. This interdisciplinary approach begins with the construction of a detailed task directive-target state dataset, which serves as the foundation for refining the capabilities of VisualBERT, a vision-language model. Through fine-tuning, VisualBERT becomes adept at accurately interpreting complex scene dynamics and anticipating the target states to achieve the task query. This is followed by the creation of an extensive knowledge graph containing important parameters of actions and objects. This knowledge base is instrumental in generating PDDL domain and problem files, taking into account both initial and target states. It plays a crucial role in the flexible subtask sequence generation for dynamic environments and tasks. Our proposed method significantly enhances the adaptability to real-world variability, thereby enabling dynamic task planning within domestic kitchen environments efficiently.

I. Introduction

The recent development of large language models, such as ChatGPT and ChatGLM [1] [2], has marked a significant leap forward in language understanding, knowledge representation, and basic reasoning, closely mirroring intelligent human behavior. This has spurred numerous researchers to incorporate these large language models into robotic manipulation, leading to developments such as RobotGPT and ROSGPT [3] [4]. A key area of research now lies in merging the scene and task understanding capabilities inherent in these rich vision-language models with the complex planning required for operational tasks.