DHS Biometric Rally 2021 results show accuracy gains, but masks and equality still challenges for some

Three quarters of the errors encountered by the U.S. Department of Homeland Security’s Science & Technology Directorate (S&T) in its latest biometrics rally occurred during the photo capture part of the process, compared to only one in four during matching.

The results of the 2021 Biometric Technology Rally, held at the Maryland Test Facility (MdTF), have been released for public review. Masks were a particular focus of the 2021 Rally.

A total of 50 different combinations of hardware devices for biometrics acquisition and algorithms for matching were analyzed, and their performance disaggregated according to gender and skin tone. The five facial image acquisition systems and ten facial recognition algorithms are anonymized with code-names, but past participants include NEC and Rank One.

DHS led 601 participants with diverse demographics through six steps, from informed consent to reporting, and analyzed the results according to anonymized vendor, but also gender and skin tone. The scenario tested for is a high-throughput identity check, meaning less than five seconds per person, without staff operation of equipment.

People without masks were correctly identified an average of 95 percent of the time, across all systems, while people wearing masks were correctly identified 86 percent of the time. The median system in the 2020 Biometric Rally identified people wearing masks 77 percent of the time.

Error rates were observed to be 3 percent higher for women than men, on average, but the most accurate system did not make any errors matching women, and the worst scores for each gender were 79 percent for men and 81 percent for women.

People with darker skin were accurately matched 93 percent of the time, compared to 97 percent for those with lighter skin, though again, the best-performing system did not make any errors matching people with dark skin.

Paravision has revealed that it is the vendor code-named ‘Salt,’ which means it was one of the vendors found to perform matches with a 100 percent true identification rate (TIR) for people of all shades of skin when their biometrics were collected with the most accurate imaging system.

Updated May 3, 2022 at 12:01pm with details about Paravision’s participation

Article Topics

accuracy  |  algorithms  |  biometric bias  |  biometric matching  |  Biometric Technology Rally  |  biometric testing  |  biometrics  |  DHS  |  facial recognition  |  Maryland Test Facility (MdTF)  |  Paravision