Improving Domain Generalization in Speech Emotion Recognition with Whisper

Transformers have been used successfully in a variety of settings, including Speech Emotion Recognition (SER). However, use of the latest transformer base models in domain generalization (DG) settings has mostly been unexplored or only weakly explored. We present here our state-of-the-art results in discrete emotion recognition across a variety of datasets, including acted and non-acted datasets, showing that Whisper is a powerful base Transformer model for this task. We show that our approach to DG with Whisper results in accuracy surpassing all previously published results, with an Unweighted Average Recall (UAR) of 74.5% averaged across the 6 distinct datasets used. We discuss some of the possible reasons behind Whisper’s superior performance to other Transformer models, though all 3 Transformer models evaluated here (HuBERT, WavLM, Whisper) show an ability to generalize as well as learn paralinguistic information successfully through fine-tuning with relatively few examples.