Modeling Vocal Entrainment in Conversational Speech Using Deep Unsupervised Learning

In interpersonal spoken interactions, individuals tend to adapt to their conversation partner's vocal characteristics to become similar, a phenomenon known as entrainment. A majority of the previous computational approaches are often knowledge driven and linear and fail to capture the inherent nonlinearity of entrainment. In this article, we present an unsupervised deep learning framework to derive a representation from speech features containing information relevant for vocal entrainment. We investigate both an encoding based approach and a more robust triplet network based approach within the proposed framework. We also propose a number of distance measures in the representation space and use them for quantification of entrainment. We first validate the proposed distances by using them to distinguish real conversations from fake ones. Then we also demonstrate their applications in relation to modeling several entrainment-relevant behaviors in observational psychotherapy, namely agreement, blame and emotional bond.