What is the FRVT?

The Face Recognition Vendor Test is an ongoing program funded by the United States. It judges the many face biometrics algorithms available to private and public entities in the nation. It is voluntary and open to anyone anywhere.

The test program has grown since 1994 to test numerous – but still focused — angles on the theme: face mask effects, demographic (including race and gender) effects and face morphing, for example. The government also posts prior tests going back to the FRVT 2000.

The goal is to give everyone an idea of what constitutes high-quality algorithm in as hard data as possible. Algorithms are named and ranked by various factors and effects like those listed above. There is no program in the world exactly like it.

What is NIST?

The National Institute of Standards and Technology created and runs it. NIST is a non-regulatory government agency and physical science lab.

Its mission is to promote U.S. industrial innovation and competitiveness by giving businesses a world-class measurement infrastructure and in so doing, create economic security and improve the quality of life of Americans.

NIST is one of the federal government’s more respected agencies is part of the Department of Commerce. The fact that the FRVT program is run by NIST and has avoided becoming a sprawling government effort has given the program credibility even with some of the writers of the lowest-ranked algorithms.

NIST, founded in 1901 and based in the state of Maryland, tests commercial technologies but it also helps create a platform of common definitions, measurements, references and standards. Its FRVT methodologies are stringent.

Operational insights

For the ongoing FRVT 1:1 Verification evaluation, large sets of face images are used to measure how commercial and academic face recognition algorithms perform. Faces that are partially hidden can be inserted to create the appearance of a subject wearing a mask.

Top performing 1:1 algorithms are measured on false non-match rate across several datasets. That rate is the proportion of mated comparisons below a threshold set to achieve the specified uniform false match rate. The false match rate is the proportion of impostor comparisons at or above that threshold.

Verification algorithms submitted since 2017 number 814, with only last year seeing more than 200 submissions. The number of unique developers is 296 with new submissions peaking at 156 last year.

The FRVT 1:N Identification total submissions, going back to 2018, is 369 algorithms, and 209 were submitted the first year. Unique developers is 105 with a comparatively steady number showing up each year.

The FRVT project looking at demographic effects used algorithms against four photo datasets collected via U.S. governmental applications: police booking photos, or mugshots; applications globally for immigration benefits; visa photographs and border-crossing photos of people entering the United States.

NIST uses this data to process 18.3 million images for 8.5 million people through 189 largely business algorithms created by 99 developers. Differing metadata accompanies each image. For instance, country of birth was considered a “reasonable proxy” for race in some circumstances.

Famously, this project found the highest false positive and rates for people of West and East African and East Asian nations. They were lowest when processing eastern European people. A smaller but just as stubborn false-positive rate difference was found comparing females to males.

FRVT limitations

This is a program without many limitations beyond budgeting. It evaluates AI algorithms, so it has little pressure to improve a product, for example. Of course, staff and managers have repeatedly demonstrated flexibility in how they judge critical aspects of facial recognition software.

The FRVT’s deliverables boil down to shepherding developers toward accurate, unbiased and standardized code. It also reports on performance and problems it encounters in individual submissions and among submissions as a group.

Algorithms analyzing data from 3D images are not included in the FRVT, whether collected through depth-sensing (like Apple’s Face ID) or video (like FaceTec).

The tests are also considered by some observers to be a better representation of biometric algorithms in a lab than imperfect, real-world data collection scenarios.

Leader

Patrick Grother, a NIST scientist, leads the agency’s biometric standards work and testing. Specifically, he manages the FRVT and related Face in Video Evaluation programs. Grother co-chairs NIST’s International Face Performance Conference on measurement, metrics and certification.

He also edits the biometrics specifications for the government’s personal ID verification credentialing program.

Article Topics

accuracy  |  AI  |  algorithms  |  biometric testing  |  biometrics  |  Face Recognition Vendor Test (FRVT)  |  facial recognition  |  NIST  |  standards