Sound method for measuring biometric bias important, tricky, and in progress

Important work on measuring biometric bias is waiting, once methods for how to measure it are worked out, says Stephanie Schuckers, director of the Center of Identification Technology Research (CITeR) at Clarkson University as well as the FIDO Alliance’s standards development director.

Schuckers gave a presentation on ‘Bias in biometric recognition: Challenges and Opportunities’ at the Authenticate 2022 event this week.

She credits Joy Buolamwini’s 2018 Gender Shades study with drawing attention to the problem, while indicting those in the media who have conflated gender classification with facial recognition.

Looking at a table from a recent NIST report that shows variabilities in false non-match rates among different demographics, Schuckers emphasized that “error rates vary strongly by algorithm,” quoting NIST.

Concerns about fairness and bias also vary, depending on the implementation.

Because FIDO is concerned with authentication, rather than identification, some concerns related to biometric bias are not central to the Alliance’s approach to password replacement. In verification scenarios, false negatives are what must be considered first, according to Schuckers, due to their impact on user experience. False positives are also important, as a security consideration, but should be a secondary consideration.

Bias should be considered at the levels of biometric capture, matching, and databases, Schuckers says.

She also played an educational video on biometric bias in access control, produced by CITeR and soon to be uploaded to its YouTube channel.

The presentation reviewed efforts to build standards around the problem, and the importance of considering system effectiveness along with equitability. That likely means setting thresholds for both overall biometric effectiveness and for low variability between different demographic groups.

The more groups are considered, the more likely a difference will be found, Schuckers says, which complicates work to measure true differentials. Some differences are likely, and some are outliers, and telling the whether a differential is isolated or systematic is a classic statistics problem.

An ad hoc group was stood up at the event, after the agenda was published, to help advance the field towards an ISO standard, currently at the working draft stage.

Article Topics

Authenticate Conference  |  biometric authentication  |  biometric-bias  |  biometrics  |  biometrics research  |  CITeR  |  FIDO Alliance  |  ISO standards  |  Stephanie Schuckers