Implementation of Multi-modal Speech Emotion Recognition Using Text Data and Audio Signals

Speech emotion detection is a difficult task, and models that employ audio characteristics in developing high-performing classifiers have received much attention in recent times. The effectiveness of emotional features extracted from speech has a significant impact on Speech Emotion Recognition's (SER) performance. On the other hand, most emotional features are sensitive to non-emotional factors such as the speaker, speaking styles, and environment. A deep dual recurrent encoder model is developed in this research. It sequentially uses text data and audio signals to comprehend speech data better and categorise it using various labels such as happy, sad, neutral, and angry. While emotional interaction comprises sound and speech, this model uses dual Recurrent Neural Networks (RNNs) to encode information from audio and text sequences and then integrate it to predict the emotion class. This architecture analyses audio signals from the signal to the language level, allowing it to use the data's information than models that only focus on audio characteristics. The model is further presented via a web application that recognises emotion from recorded spoken utterances. Cascading Style Sheets (CSS) and Hypertext Markup Language (HTML) are used to create the front-end components. On the other hand, the backend components are built with Django, SQLite as database, and Docker containers. TensorFlow, an open-source software framework for machine learning and deep neural networks were also used to create the model. Categorising utterances into one of four emotion categories, the model used on the USC-IEMOCAP dataset had an accuracy of 68.26%. This model has increasingly broadened its applications because it is an essential component of optimal human-computer interactions, such as dialogue systems.