Understanding the results of DHS S&T’s RIVTD biometrics assessment

The Remote Identity Validation Technology Demonstration, held by the U.S. Department of Homeland Security’s (DHS) Science and Technology Directorate (S&T) at the Maryland Test Facility (MdTF), provides the most detailed benchmark and breakdown available of ID document validation, face biometrics matching and presentation attack detection technologies on the market.

RIVTD’s findings can aid in understanding the state of the art in remote biometric onboarding and liveness detection. But the conditions of the research, including the three phases used and the anonymization of participating algorithm providers, may pose a barrier to that understanding.

Understanding the results is important, as the biometric technologies evaluated are principal measures in defending against identity fraud in online interactions for government services, online banking and account verifications.

S&T was supported in the tests by the Transportation Security Administration (TSA), Homeland Security Investigations Forensic Laboratory, and the National Institute of Standards and Technology.

RIVTD’s three tracks consisted of evaluations of identity document validation, face biometrics matching, and spoof detection. Participating vendors were anonymized in the results shared, but Paravision, ID R&D, Idemia and HyperVerge have each revealed their results.

ID Validation

The first track in RIVTD assessed the validation of driver’s licenses or identification cards issued by multiple U.S. states.

The tests for detecting genuine and fake IDs consisted of 1,000 real state-issued ID cards and 1,000 fakes of state-issued ID cards.

Algorithms from 12 document validation systems were tested on multiple different types of smartphones for their Document False Accept Rate (DFAR) of spoofs, and Document False Reject Rate (DFRR) for real IDs.

S&T found that document validation system performance is impacted by what state’s ID was being validated and which phone was used, leading to a recommendation that developers ensure their technology is compatible with different IDs and devices. The agency’s findings led to the recommendation that systems with error rates below 10 percent should be selected. That recommendation is built into NIST 800-63-4.

Detailed results for this track have not been publicly released.

Match to document systems

The second track measured the effectiveness of algorithms for matching selfie biometrics to ID documents. Carrying it out required S&T to create a novel test dataset.

S&T recruited 1,633 paid volunteers and the evaluation was based on over 1,000 mated comparisons and over 500,000 non-mated comparisons.

Failure to Extract Rates (FTXR) was calculated for both the document images and selfies, False Non-Match Rate (FNMR) was measured at a False Match Rate (FMR) of 1:10,000.

Six of 16 entrants to Track 2 managed to generate templates from ID documents with error rates below 1 percent, and nine delivered FNMRs below 1 percent. Five of the 16 were flagged for “large errors.”

The majority of errors found (55 percent) were in the document extraction process, and the worst system matched no selfies to ID documents, but 56 percent successfully matched people to their IDs more than 99 percent of the time. Random imposters were rejected more than 99.99 percent of the time by 63 percent of systems, but imposters matching the demographics of their target victim were 10 times more successful.

The test also showed some variability between algorithm effectiveness for different demographic groups, and for uncontrolled selfies.

PAD

The third track evaluated the ease of use and security of biometric presentation attack detection (PAD) or liveness detection technologies. S&T selected six active and 15 passive PAD systems, which it says broadly represent the state of the art.

A diverse group of 660 people was convened by S&T, and more than 1,200 presentation attacks performed.

The ease of use of the systems was measured based on error rates and time elapsed.

Among active PAD subsystems, bona fide users were validated anywhere between 41 and 94 percent of the time, with the median system successful 85 percent of the time. Error rates with active systems were 9 percent for people 18 to 45 years old, but 20 percent for people 46 and older. Passive PAD systems were processed more quickly than active ones (which tend to be videos), and also worked more consistently, with a low classification success rate of 62 percent and a median of no errors.

All attacks were stopped by two active and two passive PAD systems, prompting S&T to conclude that each can provide strong security.

One system delivered an error rate on screen and printout attacks of 88 percent, though reassuringly the median was only 2 percent.

The type of smartphone used significantly affects performance, S&T found, and error rates varied significantly across devices for both usability (or “facilitation,” as S&T calls it in results infographics) and security.

S&T plans to follow up its work on the RIVTD by running a Remote Identity Validation Rally (RIVR) throughout 2025.

Article Topics

biometric liveness detection  |  biometric matching  |  DHS S&T  |  document verification  |  presentation attack detection  |  remote identity proofing  |  Remote Identity Validation Technology Demonstration (RIVTD)