NIST seeks feedback on measurement of biometric bias

Bias in face biometrics has decreased as algorithm developers focus their efforts, according to the latest test results from the U.S. National Institute of Standards and Technology. Exactly how much they have improved, however, is difficult to even measure.

A draft of the NIST Face Recognition Vendor Test (FRVT) Part 8: Summarizing Demographic Differentials has been published for comment, with the agency seeking ways to improve its measurement of variations in biometric accuracy between groups of subjects.

Much of the document is given to explaining different statistics used to measure bias, and the advantages and limitations to different ways of doing so. The previous report on demographic differentials, FRVT Part 3, showed very low differentials among the most accurate algorithms, but significant variation overall.

There are three measures termed ‘Functional Fairness Measure Criteria (FFMC),’ suggested by John Howard, Eli Laird, Yevgeniy Sirotin, Rebecca Rubin, Jerry Tipton and Arun Vemury from the Maryland Test Facility. NIST adds two more, but finds none is perfectly suited to clearly expressing the test results.

NIST also delves into how to deal with comparisons between low error rate values. This includes challenges like how to avoid suggesting that an algorithm which returns a near-zero error rate for one demographic and a still low, but relatively higher error rate for another, is worse than an algorithm which is less accurate for both.

The ‘Max/GeoMean’ measure is identified as the leading candidate, and presented in published test results.

Vendor results are shared by NIST, with maximum false non-match rate (FNMR) and false match rate (FMR) each compared to the geometric mean rates. A result of 1 indicates matching parity. FNMR Max/GeoMean results are almost all between 1 and 2, while FMR Max/GeoMean varies all the way from slightly over 6 to over 300.

Future NIST research will apply the same kind of testing to 1:N (identification) biometric systems, and an ISO standard is in development.

Corsight claims leadership in bias reduction

Corsight AI has achieved an equal false match rate for male and female subjects between the Black and white test groups, according to a company announcement.

The company’s algorithm scored a 1.01 FNMR Max/GeoMean and a 20.63 FMR Max/GeoMean in the July 2022 update of test results.

NIST also uses algorithms from Corsight, Clearview AI, CUbox, DeepGlint, Idemia, NtechLab and Paravision as examples in a comparison of error rates with different demographics compared with Eastern European subjects.

“We’re thrilled because this is another step forward in countering claims that bias is damaging the effectiveness of facial recognition technology,” comments Corsight Chief Privacy Officer Tony Porter. “The argument that facial recognition software is not capable of being fair is frozen in time and the performance of Corsight’s latest submission demonstrates that.”

Article Topics

accuracy  |  AI  |  biometric-bias  |  biometrics  |  biometrics research  |  Corsight  |  demographic fairness  |  face biometrics  |  Face Recognition Vendor Test (FRVT)  |  facial recognition  |  Maryland Test Facility (MdTF)  |  NIST