Tailored biometric systems tests lay the foundation for trust: biometrics labs

Although language varies across regions, most regulations on biometric technology necessitate that it be robust, accurate, reliable, highly effective, or otherwise able to serve its purpose well without making egregious mistakes.

Testing makes this possible. Standardized testing frameworks such as those provided by the National Institute of Standards and Technology (NIST), FIDO and MOSIP establish a foundation for trust in the biometric industry.

But what if the tests themselves are not robust enough? Even thorough attempts to replicate real-world conditions are liable to miss the nuances of case-specific deployments. What works in a lab may fail in the field, and standard testing may not account for bias detection, attack resilience and usability under real-world conditions.

In a new Biometrics Institute report on the future of responsible biometrics, Bixelab argues that “localized and tailored testing approaches (to appropriate international standards) are essential to address demographic biases, unique vulnerabilities, and the evolving regulatory landscape.”

Tailored tests can complement standard tests to ensure biometric systems meet the unique requirements of specific demographics, devices and use cases. Testing on a broad spectrum of devices accounts for users with low-cost or legacy hardware. Attack resilience tests can gauge an algorithm’s resistance to poofing, adversarial AI attacks and other malicious attempts to compromise its integrity. And throwing real-world variables at an algorithm can assess its ability to function in dynamic lighting, humidity or background noise conditions.

“Biometric systems often interact with diverse populations and operate in varied conditions. Without localized testing, organizations risk deploying systems that inadvertently exclude certain groups or fail to perform reliably in specific regions.”

Diversity is strength for training facial recognition systems

The direct claim on demographic diversity argues that “certain facial recognition systems have been shown to perform poorly on darker skin tones due to limited diversity in training datasets.” (Though in NIST’s latest testing, the most accurate algorithms show very low differentials.) The issue is partly one of pace: the ecosystem has grown rapidly, and certain components have been fragmented. Information sharing between developers is limited.

And most importantly, Fime says, “there is now consensus that an AI model is only as good as the data that trained it.” The Convolutional Neural Networks (CNNs) that enable deep machine learning based on face biometric templates for facial recognition systems need diverse reference material; without it, “the algorithm’s performance will be inadequate.”

“This is bias,” Fime says. “It is not necessarily malicious, but it is a reality” – and it needs “urgent attention across the biometric value chain; from software and sensor developers to device OEMs and standardization bodies and governments.”

Fime says that, “to know the best metric for evaluating the potential bias of a biometric system, it is important to measure the effectiveness of each metric on the specificity and accuracy of the system. With this in mind, leading biometrics experts have created an innovative method to inject bias on biometric systems and evaluate the effectiveness of each metric. This system allows experts to inject selective biases for specific demographic sub-groups, with control over the strength of each bias. Exercising direct control over the strength of each bias allowed the expert to monitor the effectiveness of each metric against known variables.”

Know your biometric tests

The more you test, the more you know, and Ingenium outlines the differences between performance testing, security testing, bias and fairness testing, and regulatory and compliance testing.

Performance testing “measures how well a biometric system functions in real-world scenarios.” It includes False Acceptance Rate (FAR) and False Rejection Rate (FRR) – metrics that indicate, respectively, how often a system mistakenly accepts an imposter or rejects a legitimate user – and environmental testing to assess system performance under various material conditions.

Security testing, “is essential to identifying vulnerabilities in biometric systems and protecting against malicious attacks.” It encompasses spoofing and presentation attack detection (PAD) to evaluate detection capabilities for fake fingerprints, 3D-printed faces or voice mimicry; ensuring secure encryption of databases and compliance with regulations; and simulating real-world fraud tactics, including deepfakes, synthetic media and injection attacks.

Bias and fairness testing “ensures that biometric systems work equally well for all demographic groups, and is “crucial for ethical implementation.” Diversity assessments, error rate analysis across demographics and algorithmic audits support this process.

Regulatory and compliance testing measures systems against ethical, legal and regulatory frameworks like GDPR, the Illinois Biometric Information Privacy Act (BIPA) and the AI Act.

“Independent laboratories and testing facilities play a crucial role in evaluating biometric systems before they reach the market,” Ingenium says. NIST conducts “rigorous biometric testing programs, such as the Face Recognition Vendor Test (FRVT) and the MINEX evaluation for fingerprint systems.” The International Organisation for Standardisation (ISO) provides biometric testing standards, such as ISO/IEC 19795 Performance testing framework for biometric systems and ISO/IEC 30107 PAD standards.

And industry bodies that provide evaluations and certification, such as the FIDO Alliance, also play a role.

All of this is important, according to Ingenium, because “the future of responsible biometrics hinges on rigorous testing, ethical considerations, and compliance with security and privacy standards.”

Idiap report notes ‘critical need for equitable and trustworthy FR systems’

A new research report from the Idiap Research Institute “systematically explores the issue of demographic bias in FR through different yet interrelated sections: causes, datasets, assessment metrics, and mitigation strategies.”

Causes identified begin with “imbalanced or unrepresentative datasets,” but also factor in variations in skin tone, “the sensitivity and limitations of algorithms in addressing demographic attributes,” and image quality.

Co-authors Ketan Kotwal and Sébastien Marcel of IEEE write that, “by categorizing key contributions in these areas, this work provides a structured approach to understanding and addressing the complexity” of bias in facial recognition systems.

Ultimately, they emphasize the “critical need for equitable and trustworthy FR systems.”

New biometric testing lab opens in Moscow

The need for testing is global, as evidenced by the new unified biometric testing center (ECBI) operating in Moscow, a collaboration between the Moscow Department of Information Technology and the Russian Biometric Society non-profit partnership. A note in Oreanda News says the center will be “engaged in research, piloting and evaluating solutions using video analytics.”

According to Moscow Deputy Mayor Natalia Sergunina, the city is actively studying best practices for computer vision technologies and “implementing them to solve urban problems.”

Article Topics

accuracy  |  biometric bias  |  biometric testing  |  biometrics  |  Biometrics Institute  |  BixeLab  |  Fime  |  Idiap  |  Ingenium Biometrics