Face biometrics vendors Oosto and FaceTec identify limitations to NIST FRVTs

The leading benchmark for face biometrics is not well-suited to evaluating technology for video surveillance use cases, Oosto argues in an article posted on its website.

It is just the latest example of a vendor pointing out the limitations of the highly-regarded Face Recognition Vendor Test (FRVT) from the U.S. National Institute of Standards and Technology, following a similar line of recent commentary from FaceTec.

In ‘5 Limitations of NIST’s FRVT Testing for Video Surveillance,’ Oosto CMO Dean Nicolls begins by explaining what the test is and the importance of NIST as “an independent body that is testing third-party software – a la Consumer Reports or JD Power – and providing industry performance benchmarks.”

Enterprises should be aware, however, that the FRVT is not applicable to video surveillance software, he argues.

Nicolls points out that the test is intended for forensic applications, rather than real-time identification. Function in real-world conditions is also critical for video surveillance, but even the ‘wild’ category in NIST’s test is not equivalent to the kind of images processed by video surveillance systems.

“When I use the term ‘in the wild,’ this is different from NIST’s definition of ‘wild’ where people are not necessarily posing or cooperating for a photo,” Nicolls writes. “NIST’s wild images may be blurry, or of low quality, but the images are still relatively clear, the camera is generally at face level, and the lighting is good.”

FRVT also does not evaluate the ability to pick an individual out of a crowd, consider video camera quality, or test images captured at extreme angles.

The purpose of the post is not to criticize NIST, Nicolls insists, but rather to help enterprises and technology buyers understand the limits of the influential benchmark.

Oosto encourages NIST to create a new testing methodology to handle the particular conditions and requirements of facial recognition in video surveillance solutions.

Echoes of FaceTec PAD commentary

The argument generally tracks with arguments made by FaceTec in its ‘NIST FRVT-PAD Commentary’ submitted by the company in late-April.

The commentary was submitted to NIST in response to a request for comments on its proposed FRVT-PAD (presentation attack detection) framework. While addressing the PAD assessment proposal, the commentary makes points similar to Oosto’s in that they suggest a range of technologies the test will not adequately address.

FaceTec says that the PAD standard proposed as a starting point “has been outdated for years,” referring to the ISO/IEC 30107-3 publication in 2017.

Shortly after the commentary was published, FaceTec announced the development of a 3D face biometrics model that it said delivers a step-change in accuracy, with a false acceptance rate of 1 in 125 million. The algorithm cannot be tested as part of the FRVT, however, which only uses 2D images.

That limitation means that NIST will be unable to assess Apple’s Face ID, as well as FaceTec’s biometrics.

Defense against deepfake video injections should also be included in any test, FaceTec argues. The commentary goes on to explain bypass attacks, and other vulnerabilities, including in-device sensors.

Article Topics

biometric testing  |  face biometrics  |  Face Recognition Vendor Test (FRVT)  |  FaceTec  |  facial recognition  |  NIST  |  Oosto  |  real-time biometrics  |  standards