Interviewee Performance Analyzer Using Facial Emotion Recognition and Speech Fluency Recognition

Analysis of the performance of an interviewee is a complex and challenging task. Our proposed system automates this process by building two multiclass classification models. Video captured during an interview is given to the proposed system which extracts frames and Audio from it. Frames are given to the first model which is a facial emotion recognition model it uses HaarCascade classifier, Gabor filters, and Convolution Neural Network for classification of facial emotion as one of these seven emotions like Happy, surprise, angry, disgust, neutral, fear, sadness. Audio is given to the second model which uses Mel frequency cepstral coefficient features and logistic regression for speech classification as four classes Fluent, Stuttering, Cluttering, and Pauses. Predictions of these two models can be combined to give a performance rating for the interviewee. Compared to only CNN based and Deep Neural Network based facial emotion recognition, the Gabor Filter based approach which we have used gave better accuracy with a smaller number of hidden layers and less training time.