Speaker Normalization for Self-Supervised Speech Emotion Recognition

Large speech emotion recognition datasets are hard to obtain, and small datasets may contain biases. Deep-net-based classifiers, in turn, are prone to exploit those biases and find shortcuts such as speaker characteristics. These shortcuts usually harm a model’s ability to generalize. To address this challenge, we propose a gradient-based adversary learning framework that learns a speech emotion recognition task while normalizing speaker characteristics from the feature representation. We demonstrate the efficacy of our method on both speaker-independent and speaker-dependent settings and obtain new state-of-the-art results on the challenging IEMOCAP dataset.

1. INTRODUCTION

Over the last decade, advances in speech processing have been extremely impressive. Challenges like personal voice assistants that seemed daunting merely ten years ago are now part of our day-to-day routine. A crucial part of those systems is to interact with the user and understand his intents and emotions. Understanding the user’s emotion can build trust between the user and the system and improve the user experience. Improving the accuracy of speech emotion recognition has, therefore, been a major research focus in recent years.