Measuring IDV performances at Authenticate

For relying parties deciding on an identity verification (IDV) vendor, what are the key metrics to look at to understand which is the best fit? Christopher Cortes, Product Manager at Onfido and the Vice Chair of the FIDO Alliance Identity Verification Working Group, spoke at the recent Authenticate conference about best practices for measuring and assessing the performance of IDV systems.

In his talk, Cortes presents a scenario in which a relying party expanding into a new territory is in need of IDV services, and is presented with three different approaches to measuring performance. He says consistency is key, as is keeping in mind regional scope in terms of what is most important to the relying party.

“You want to make sure they are testing against something that you care most about,” Cortes says. “Also make sure that, in terms of coverage, the actual attack types that are out in that market are being covered. That level of consistency will bring comparability” in developing a standard set of measurements.

Compliance is also a clear yardstick for relying parties, Cortes says. “There are many different compliance standards all across the globe, and it is really important that a relying party is going to look at how they can showcase compliance to regulators.” Recognized standards in different regions can help point to regional strengths, but ongoing monitoring is necessary to factor in changes in standards over time. “Compliance isn’t just a static thing.”

The same goes for operations. Cortes emphasizes the need to maintain operational standards in a real-world scenario, to address questions like, “What is production doing? How is production performing against live volumes?” This can help identify new attack vectors and top impacting issues, which can ideally be shared and compared with peers in the sector to work toward collective goals.

Also on Cortes’ list of must-have is improving product performance to isolate top effects and root causes, and to measure their impact on IDV. “This may also involve creating new content to test against your systems,” he says.

Third party measurement can help ensure vendors have the region-specific validation required for specific geographies. Positive feedback loops can expedite the resolution of customer issues, such as false rejections.

A Production Results Review, which can be costly and requires fraud detection expertise, reviews the analysis of what a production system has already performed. It also presents the opportunity to generate ground truth data, a known set of inputs that have been run against your existing production system, and use it in a ground truth dataset to test against arising issues and irregularities.

In terms of actual metrics being proposed, performance against a wide range of document attacks including digital and physical tampering, procedural, and liveness is categorized into tiers for Simple, Advanced, Expert and State. Cortes posits a five-tier pyramid classifying document sophistication, and a harmonized language on document validation metrics for classification accuracy, extraction correctness, false acceptance rate, false rejection rate and approval rate.

Article Topics

Authenticate Conference  |  biometrics  |  FIDO Alliance  |  identity verification  |  Onfido  |  standards