Robustness Analysis uncovers Language Proficiency Bias in Emotion Recognition Systems

Emotion recognition in conversations (ERC) has rapidly emerged as a vital instrument in enhancing human-computer interactions. However, concerns about the fairness and biases of these ERC systems persist and remain to be addressed by assessing their robustness. This study presents a methodology to analyze the robustness and bias of an ERC system by including complexities of user input with varying English language proficiency. We develop a novel, hybrid approach to create text perturbations by combining natural language generation techniques with rule-based constraints to simulate language proficiency levels. Specifically, we utilize the capabilities of GPT-3 to generate text modifications based on language proficiency characteristics introduced by the internationally recognized Common European Framework of Reference (CEFR). Based on the application of the widely-used COSMIC model, our robustness analysis discloses that the ERC system’s performance decreased as language proficiency diminished. Hence, this study demonstrates the presence and implications of language proficiency bias in ERC systems, resulting in discriminatory consequences for non-native English speakers. Overall, our perturbation exhibits versatility for diverse analysis objectives. For instance, it allows investigating gender bias and examining unilateral linguistic bias involving native and non-native speakers. By making our implementation publicly accessible, we aim to foster the advancement of fair ERC systems.