A Novel Multimodal Situated Spoken Dialog System for Human Robot Communication in Emergency Evacuation

Given the need for multimodal autonomous human-robot interactive systems serving complex situations, a noise-robust and context- aware situated multimodal spoken dialogue system (MSSDS) in emergency evacuation missions is presented. The MSSDS system is composed of: (1) a context and noise robust Wake-Up-Word (WUW) (a) to initiate the dialogue, and (b) to detect the context switching by explicit use of WUW during emergency evacuation mission steps, (2) multi-turn user- initiated interactive text-and-spoken dialog communication system, and (3) an interactive voice and text interface for human-robot communication designed based on dialogues used in real-world emergency situations. In an emergency environment, speech is mixed with different noises, and therefore communication using speech in such an environment is challenging. We handle the noise by using Team Connect Ceiling (TCC) beam-forming microphone arrays. Innovative and useful applications of spoken dialogue systems, presented as proof of concept, constitute another contribution. Numerous digital assistants, keyword spotting, and wake-up-words-based technologies have already been developed, but these are mainly used indoors. Our objective is to support communication in complex environments, e.g., indoors and outdoors, in human and machine teaming, via a wake-up-word-based multimodal interactive system. The development of the real-world application to communicate with the robot using multimodalities in complex situations based on the presented approach is in progress while the presented simulated approach is reflecting parts of this development. Numerous machine learning technologies and toolkits have been applied in this ongoing development process. The novelty of WUW-based MSSDS is discussed in this paper. Our Markov Decision Process (MDP) evaluation shows that the WUW-based MSSDS performs better.