Meta Balanced Network for Fair Face Recognition

Although deep face recognition has achieved impressive progress in recent years, controversy has arisen regarding discrimination based on skin tone, questioning their deployment into real-world scenarios. In this paper, we aim to systematically and scientifically study this bias from both data and algorithm aspects. First, using the dermatologist approved Fitzpatrick Skin Type classification system and Individual Typology Angle, we contribute a benchmark called Identity Shades (IDS) database, which effectively quantifies the degree of the bias with respect to skin tone in existing face recognition algorithms and commercial APIs. Further, we provide two skin-tone aware training datasets, called BUPT-Globalface dataset and BUPT-Balancedface dataset, to remove bias in training data. Finally, to mitigate the algorithmic bias, we propose a novel meta-learning algorithm, called Meta Balanced Network (MBN), which learns adaptive margins in large margin loss such that the model optimized by this loss can perform fairly across people with different skin tones. To determine the margins, our method optimizes a meta skewness loss on a clean and unbiased meta set and utilizes backward-on-backward automatic differentiation to perform a second order gradient descent step on the current margins. Extensive experiments show that MBN successfully mitigates bias and learns more balanced performance for people with different skin tones in face recognition. The proposed datasets are available at http://www.whdeng.cn/RFW/index.html.

1 Introduction

Recently, with the emergence of deep convolutional neural networks (CNN) [1], [2], [3], [4], [5], research focus of face recognition (FR) has shifted to deep-learning-based approaches [6], [7], [8] and the accuracy was dramatically boosted to above 99.80% on the Labeled Faces in the Wild (LFW) dataset [9]. However, the recognition accuracy is not only aspect to attend when designing learning algorithms. As a growing number of applications based on FR have integrated into our lives, its potential for unfairness is raising alarm. For example, Amazon’s Rekognition Tool incorrectly matched the photos of 28 U.S. congressmen with the faces of criminals, especially the error rate was up to 39% for Black faces. According to these reports [10], [11], FR system seems discriminative based on classes like race, demonstrating significantly different accuracy when applied to different groups. Such bias can result in mistreatment of certain demographic groups, by either exposing them to a higher risk of fraud, or by making access to services more difficult. Consequently, there is an increased need to guarantee fairness for automatic systems and prevent discriminatory decisions.