A Two-Stage Hierarchical Multilingual Emotion Recognition System Using Hidden Markov Models and Neural Networks

Speech emotion recognition continues to attract a lot of research especially under mixed language speech. Here, we show that emotion is culture/language dependent. In this paper, we propose a two-stage emotion recognition system that starts by identifying the language then using a dedicated language-dependent recognition system for identifying the type of emotion, The system is able to recognize accurately the four main types of emotion, namely Neutral, happy, angry, and sad. These types of emotion states are widely used in practical setups. To keep the computation complexity low, we identify the language using a feature vector consisting of energies from a basic wavelet decomposition of the speech signal. The Hidden Markov Model is then used to track the changes of this energy feature vector to identify the language achieving recognition of accuracy close to 100%. Once the language is identified, a set of traditional speech processing features including pitch, formats, MFCCs.... etc, are used with a basic Neural Network architecture to identify the type of emotion. The results show that that identifying the language first can substantially improve the overall accuracy in identifying emotions. The overall accuracy achieved with the proposed hierarchical system was above 93 %. The work shows the strong correlation between language/culture and type of emotion, and can further be extended to other scenarios such as gender-based recognition, facial-expression based recognition, age-based recognition ... etc.