Generalization of Self-Supervised Learning-Based Representations for Cross-Domain Speech Emotion Recognition

Self-supervised learning (SSL) from unlabelled speech data has revolutionized speech representation learning. Among them, wavLM, wav2vec2, HuBERT, and Data2vec have produced benchmark performances on automatic speech recognition. However, few studies have explored the generalization of SSL-based representations to different tasks based on paralinguistic information in speech such as emotion recognition. This paper explores the generalization of all four popular SSL models for speech emotion recognition (SER) when trained and tested in different domains. We aim to understand how adaptable these SSL representations are when using simple domain adaptation techniques. The evaluation considers emotional speech databases that deviate in language, recording conditions, and emotional distribution, providing very different target domains. The results reveal the necessity to fine-tune the representations for the SER downstream. As the differences between the source and target domain increase, we observe that the unsupervised domain adaptation techniques are more effective. The analysis in this study provides useful insights to understand the advantages of different representations for domain adaptation in SER.

1. INTRODUCTION

A significant breakthrough in the field of speech processing is the use of self-supervised learning (SSL) as an appealing strategy for harnessing insights from extensive unlabeled datasets. SSL-based solutions have demonstrated their effectiveness in enhancing downstream tasks such as automatic speech recognition (ASR) [1]–[7]. These models have been successfully used in other speech tasks, including speech emotion recognition (SER) [8]–[10]. Recognizing emotions from speech plays a major role in natural human-computer interaction [11], [12], so it is important to improve SER solutions before they are deployed in practical applications. Even though recent studies in SER showed significant gains by using SSL-based models for an SER task, very few studies have explored their performance in domain conditions that differ from the ones used for training the models. One of the major barriers to obtaining a deployable SER system is its generalization across different domain conditions. Hence, it is necessary to assess the SSL-based model’s performance across different domain conditions.