Dialog Breakdown Detection using Multimodal Features for Non-task-oriented Dialog Systems

In this paper, we propose a dialog breakdown detection method for non-task-oriented dialog systems, using multimodal information such as language information, acoustic information, facial expressions, and gaze behavior. Existing automated dialog systems do not always respond appropriately to the utterances of their users, and these systems lack the ability to restore the flow of the dialog by noticing when they have made a serious error. To address this problem, in this study we propose a method of detecting dialog breakdowns using multimodal information collected from users. Data from interactions between human users and the dialog system, such as the content and acoustic features of the user’s speech, as well as their facial expressions and gaze direction, are collected using the‘ Wizard of Oz’ (WOZ) method. We then focus on exchanges where breakdowns in the dialog occurred. Features are extracted from the data for each modality (language, acoustic and visual). After dimension compression using principal component analysis (PCA), dialog breakdown detection is conducted using a deep neural network (DNN). In our evaluation experiment, the proposed method achieved an 85.7% accuracy rate for dialog breakdown detection.