The Importance of Qualitative Elements in Subjective Evaluation of Semantic Gestures

Gestures play a vital role in face-to-face interactions, from conveying speaker attitudes to relaying information not present in speech. Gestures have been widely shown to be linked to the meaning, form and timing of co-speech context of their production. Producing convincing, relevant, and informative semantic gestures is an ongoing challenge in the field of gesture generation for embodied conversational agents. In this paper, we put forward a novel technique to select semantically-related gestures from a gesture database, and present two experiments which highlight the importance of measuring the qualitative impact of semantically-related gestures on the viewer. In the first experiment, we demonstrate a strong correlation between subjective perception of energy level of the speaker and perception of the semantic relatedness of the co-speech transcript. In the second experiment, we attempt to measure semantic information conveyed in gesture on specific qualitative dimensions. We then discuss the implications and impacts of these findings, including the limitations of the strength of claims we can make when using the original gesture that accompanies an utterance as an evaluative baseline.

I. Introduction

Gestures are powerful non-verbal behaviors that influence viewer perceptions about the speaker [38], [27], qualitatively change the way viewers understand information [24], and provide a window into the cognitive processes of the speaker [10]. Virtual agents that use appropriate gestures are seen as more persuasive [20], natural, and humanlike [40]. Work in both behavioral psychology and gesture generation suggests that gestures may be composed into meaningful components, capable of conveying complex semantic messages using a limited repertoire of motions [5].