Idiap researchers propose metric for measuring biometrics fairness

A pair of researchers from the Idiap Research Institute have developed a metric to use in assessing the fairness of biometric systems, or its lack in providing disparate results in matching people based on gender and skin color.

The paper by Idiap’s Tiago de Freitas Pereira and Sébastien Marcel on ‘Fairness in Biometrics: a figure of merit to assess biometric verification systems’ describes the use of reference databases which are intended to represent operational conditions in the benchmarking stage of the machine-learning development pipeline.

The accuracy shown in those benchmarking processes, however, often differs between demographic groups, giving rise to fairness concerns.

The researchers discuss the factors which could go into considering a biometric system to be fair, and propose the use of fairness discrepancy rate (FDR) as a measurement of differences in accuracy. The 11-page paper also includes a case study of FDR using facial recognition.

“Most of the works in the biometrics community assess fairness in verification systems by comparing DET curves, and/or ROC curves of different demographic groups separately,” the report authors write. “This type of comparison assumes that decision thresholds are demographic-specific, which is not feasible in operational conditions and doesn’t proxy statistical separation. FDR addresses that by assessing demographic discrepancies assuming single decision thresholds.”

The source code, trained models, and scores are also made publicly available to enable other to reproduce the work.

Much of the paper details the formula for determining FDR, and arrives at a metric which they say indicates fairness the closer the “area under FDR” is to 1. In an example, they call a finding of 0.999 “fair,” and one of 0.777 “unfair.”

Unsurprisingly, in tests of several systems against three public datasets, some significant imbalances are identified.

Work on the related issues of bias and fairness in biometric algorithms and datasets continues across the industry, even as policy-makers attempt to get a fix on the problem.

Article Topics

accuracy  |  biometric dataset  |  biometrics  |  biometrics research  |  demographic fairness  |  facial recognition  |  identity verification  |  Idiap  |  machine learning