Mitigating Demographic Bias in Face Recognition via Regularized Score Calibration

Demographic bias in deep learning-based face recog-nition systems has led to serious concerns. Several ex-isting works attempt to mitigate bias by incorporating demographic-specific processing during inference, which requires knowledge or learning of demographic attribute with an additional cost. We propose to regularize training of the face recognition CNN, for demographic fairness, by im-posing constraints on the distributions of matching scores. Our regularization term enforces the score distributions from different demographic groups to respect a pre-defined probability distribution, as well as it penalizes misalign-ment of distributions across demographic groups. The pro-posed method improves fairness of face recognition models without compromising the recognition accuracy, and does not require extra resources during inference. Our experi-ments indicate that in a cross-dataset testing, the regular-ized CNN can reduce the variation in accuracies (i.e., more fairness) of different demographic groups up to 25% while slightly improving recognition accuracy over baselines.