Enhancing Augmented Reality Dialogue Systems with Multi-Modal Referential Information

In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual environments. We construct the “SIMMC2-Point” dataset, which transforms the original SIMMC2 dataset from virtual reality (VR) into AR environments, highlighting the additional introduced pointing modality to support understanding user's multi-modal intentions in AR. By harnessing the power of BART and CLIP models, we design the architecture of multi-modal dialogues that effectively captures spatial and attribute information. Then, a series of ablation experiments based on our designed SIMMC2-Point dataset and dialogue models underscores the significance of the pointing modality in enhancing the performance of dialogue systems across various tasks. Our work represents a crucial step forward in AR dialogue systems, facilitating seamless interactions within immersive virtual environments.