Classification of Complex Information: Inference of Co-Occurring Affective States from Their Expressions in Speech

We present a classification algorithm for inferring affective states (emotions, mental states, attitudes, and the like) from their nonverbal expressions in speech. It is based on the observations that affective states can occur simultaneously and different sets of vocal features, such as intonation and speech rate, distinguish between nonverbal expressions of different affective states. The input to the inference system was a large set of vocal features and metrics that were extracted from each utterance. The classification algorithm conducted independent pairwise comparisons between nine affective-state groups. The classifier used various subsets of metrics of the vocal features and various classification algorithms for different pairs of affective-state groups. Average classification accuracy of the 36 pairwise machines was 75 percent, using 10-fold cross validation. The comparison results were consolidated into a single ranked list of the nine affective-state groups. This list was the output of the system and represented the inferred combination of co-occurring affective states for the analyzed utterance. The inference accuracy of the combined machine was 83 percent. The system automatically characterized over 500 affective state concepts from the Mind Reading database. The inference of co-occurring affective states was validated by comparing the inferred combinations to the lexical definitions of the labels of the analyzed sentences. The distinguishing capabilities of the system were comparable to human performance.

1 Introduction

Affective states and their behavioral expressions, and in particular their nonverbal expressions in speech, are important aspects of human reasoning, decision making, and communication [1], [2], [3], [4]. According to the “Theory of mind” [5], [6], affective states such as beliefs, intents, desires, pretending, and knowledge can be the cause of behavior and thus can be used to explain and predict others behavior. The integration of affective states and their behavioral correlates in fields such as human-computer interfaces and interactions (HCIs), human-robot interactions (HRIs), and speech technologies can enhance the system and user performance and has many potential applications [3], [4], [7]. Therefore, there is an increased interest in detecting, analyzing, and imitating these cues.