Emotion Recognition by Facial Expressions and Speech Using Deep Learning

A highly harmonic human-computer interaction experience is being realized thanks to the quick development of emotion recognition. This research provided two ways for emotion recognition, one method using facial expressions and another model using speech, when considering the complementary of the emotional content of speech and expressions on the face. We use the CNN to learn facial emotion features and LSTM to acquire speech emotion features. Concurrently the multiple small-scale kernels with a convolution block will be drafted to extract aspects of facial expression. We use the LSTM model drafted to extract the significant speech features. Finally, we use aspects of facial expression and speech features separately to find the emotion. The model for recognizing facial emotions was evaluated using the FER 2013 dataset and the model for voice emotion recognition was tested on the RAVDESS dataset.