NIST testing shows accuracy of face biometric algorithms developed pre-pandemic lowered by masks

Biometric facial recognition algorithms developed since the beginning of the COVID-19 pandemic may be able to identify people wearing masks better than legacy algorithms, but even the best ones developed previously do so “with great difficulty” according to the latest testing from the U.S. National Institute of Standards and Technology.

NIST tested 89 commercial facial recognition algorithms for the ‘Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms.’ In testing with digitally applied face masks, even the best algorithms had error rates between 5 and 50 percent in one-to-one matching.

The research team came up with nine mask variants in black or the blue of surgical masks, some just covering the ‘wearer’s’ mouth and nose, while some cover the whole lower face.

NIST Computer Scientist Mei Ngan, one of the report’s authors, says the test was motivated by the pandemic, and the organization plans to test new algorithms developed with masks in mind later in the summer.

“We can draw a few broad conclusions from the results, but there are caveats,” Ngan notes. “None of these algorithms were designed to handle face masks, and the masks we used are digital creations, not the real thing.”

The accuracy of the algorithms declined substantially in every case. While the top performing algorithm with unmasked faces failed about 0.3 percent of the time, their failure rate was closer to 5 percent for faces occluded with the digital masks. Many “otherwise competent” algorithms failed between 20 and 50 percent of the time. Masked images also caused algorithms to be unable to process the face at all, due to a “failure to enroll or template.”

How much of the wearer’s nose is exposed greatly affects accuracy, according to comparisons NIST did with low, medium, and high levels of nose coverage. The shape and color of the mask also make significant differences, with blue masks or more rounded ones degrading performance less. Time and resource constraints made NIST unable to test the effect of color in great depth.

More encouragingly, while false negatives increased, false positives were stable, or even declined somewhat.

DHS recently expressed concern that its facial recognition systems may be less effective with people wearing masks in a bulletin. Companies claiming to have developed algorithms that can handle masks with high accuracy include Innovatrics and SAFR, while other like FacePhi have developed algorithms based on periocular biometrics.

Future tests, in addition to evaluating newer algorithms, will also test for one-to-many matches and other variations to broaden the insights provided.

Ngan says that accuracy is expected to improve, but that the results are consistent with pervious FRVT evaluations.

“Users should get to know the algorithm they are using thoroughly and test its performance in their own work environment,” she says.

Article Topics

accuracy  |  algorithms  |  biometric identification  |  biometrics  |  facial recognition  |  NIST  |  testing