A Machine Learning and Deep Learning based Approach to Generate a Speech Emotion Recognition System

The most common and general medium via which we humans convey or communicate our thoughts, emotions, feelings or ideas artlessly is by speech or articulation. Blending of this artless way of speech with the technological advancements of AI, has given rise to the importance of building emotion recognition systems from speech today. Even more, the speech/articulation emotion recognition system presented here is also to contribute in and facilitate various emerging applications of today like, in detecting persons’ physiological state (as in lie detectors), also be used in forensics, medicine. The proposed work identifies/associates an appropriate label/emotion for the respective emotion from speech presented in the form of an audio file (.wav format). About 4240 audio samples are taken. 1440, 2800 samples from RAVDESS and TESS datasets are considered respectively. After this process of Data collection, features are separately extracted for each audio dataset mentioned above. Energy, pitch, ZCR, co-efficient of Mel frequency ceptrum (MFCC) are some of the features considered in this study. Furthermore, clubbing and merging of 2 datasets is performed resulting in a total of 4240 rows and 24 columns (features/characteristics including 1class label) of audio samples. The resulting 4240 samples of feature dataset is split/bifurcated into training and testing set by considering 3 different possibilities/instances viz; 60%-40% ratio, 70%-30% ratio, 80%-30% ratio. The models namely CNN, Random forest and Support Vector Machine are trained to classify the dataset into 8 different emotions (neutral, calm, happy, sad, angry, fearful, disgust, surprise). An attempt to implement the models using two very essential disciplines of AI i.e. Machine Learning and Deep Learning is made here. The accuracy or results are depicted by generating confusion matrices on test data for CNN, RF and SVM models (Each model is trained and test across 3 different ratios viz; 60%-40%, 70%-30%, 80%-20%). Considering all eight emotions/ class labels, the accuracy resulted by the CNN model is higher when compared to the other models RF, SVM. As an example, taking class “neutral” into account, about 98% accuracy was obtained using CNN unlike SVM and RF wherein accuracies are 97% and 95% respectively. The above mentioned observations are for 60%-40% splitting ratio while building the model. The other observations are tabulated in the later sections of this article accordingly.