PoSE: Suppressing Perceptual Noise in Embodied Agents for Enhanced Semantic Navigation

Embodied agents navigating unknown environments face the challenge of optimizing exploration based on semantic information. Conventional methods, reliant on collected data or pre-defined rules, have limitations in scalability and applicability, while pretrained language models based methods focusing on textual modality encounter perceptual noise, which affects decision-making. To mitigate these problems, this paper presents Prompt-based Vision Context Semantic Exploration (PoSE), an innovative method that leverages prior knowledge from vision-language models (VLMs) to suppress perceptual noise. Through its unique design of prompts that use existential logic, PoSE reduces misidentifications of target objects within the observed environment. It also introduces a unique exploration map to translate target locations into exploration coordinates. The proposed methodology is evaluated on the ALFRED benchmark, demonstrating improved performance against previous rule-based and task-specific data-driven exploration policies. Furthermore, PoSE's semantic exploration policy offers enhancements over pretrained language-model based exploration methods that focus on text modality, showcasing its effectiveness and generality.