Deep independent audio-visual affect analysis

Induced affect the emotional effect of an object on an individual can be quantified through two metrics: valence, which quantifies how “positive” or “negative” something is, and arousal, the emotional intensity from “calm” to “exciting”. Affective content analysis of visual media is a challenging problem due to differences in perceived reactions. The best affect-annotated video data sets are often analyzed by feeding large amounts of visual and audio information through machine-learning algorithms. The goal is to maximize accuracy, with the hope that each representation will bring useful information to the table. We quantify how different modalities such as visual and audio information can aid in understanding affect. To that end, we train independent models for visual and audio modalities. Each are convolutional neural networks paired with support vector machines to classify valence and arousal. We also train various ensemble models that combine multi-modal information. We find that our visual network alone achieves state-of-the-art valence classification accuracy and that our audio network, when paired with our visual features, achieves competitive results on arousal classification. Each network is much stronger on one metric than the other. This may lead to more sophisticated approaches to accurately identify affect in visual media.