Convolutional Neural Networks for Image Emotion Recognition by fusing Differential and Supplementary Information

Emotions arise out of complex phenomena, which are believed to have a biological basis. Neuroscience research has demonstrated that emotions are related to distinct patterns of brain activity and the release of certain chemicals, such as hormones and neurotransmitters, into the bloodstream. Emotion recognition is used in many applications, such as advertising, social networking, and cinema. In this paper, we propose a deep convolutional neural network (CNN) fusion technique made up of two parts: a differential-CNN system to extract emotional features from an image and combine them using convex combination and a supplementary CNN to extract central object details of an image and combine it with differential-CNN features. We amplify the difference between minute latent representations of an image via differential-CNN features. The type of emotion an image elicits is highly influenced by its central object. Hence supplementary CNN helps supplement the differential-CNN features to improve emotion recognition. In comparison to the contemporary state-of-the-art methods, our proposed method showed an absolute gain of 6% on the primary dataset and outperformed them on the different secondary datasets social media web scraping process.

I. Introduction

Emotions are strongly related to the activation of particular brain regions and the release of specific hormones and neurotransmitters into the bloodstream. For instance, the amygdala, a little almond-shaped structure in the brain, initiates the release of adrenaline and other stress chemicals that prepare the body to respond to a perceived threat when we experience fear. Research on emotion analysis has grown significantly in the past two decades. Previous attempts to solve this challenge used handcrafted methods based on psychological ideas [1] and image perception. Color, edges, lines, texture, composition and picture descriptors like Histogram of Oriented Gradients (HOG) were handcrafted. Balance, emphasis and harmony were linked to the emotions elicited from an image. Lu et al. [2] attempted to retrieve shape features and classify the emotions of an image. Yanulevskaya et al. [3] retrieved Gabor and Wiccest surface texture features from images and mapped them to emotions. The type of line influences the emotional response as well. In the image, oblique, horizontal and vertical lines had distinct effects on the emotional response. Chang et al. [4] extracted edges, ridges and lines to classify images. Zhao et al. [5] extracted features using image descriptors (such as Histogram of Oriented Gradients) and combined them with other handcrafted features. Both Mehrabian and Valdez [6] attempted to map color primitives to emotions and classify images.