FB pixel

IPVM launches evaluation of facial recognition systems in real-world conditions

IPVM launches evaluation of facial recognition systems in real-world conditions

Security camera trade publication IPVM has published its first-ever Facial Recognition Rankings, which indicate significant differences between the performance claimed by vendors and the accuracy of their commercial systems in real-world conditions.

The evaluation included tests of the various commercial systems in a range of scenarios. Each vendor was assessed for performance on production at different angles of camera tilt and subject approach, with subjects wearing accessories, in groups, and captured with low, normal, and infrared lighting. Rankings are provided in each category, along with information on usability and installation. IPVM plans to expand the evaluation in future editions.

Biometric enrollment and identification were assessed using the NIST IJB-C dataset.

The surveillance camera compared facial recognition performance by commercially available cameras from Avigilon, BriefCam, Hikvision, IronYun, Oosto, Rhombus, Turing and Verkada, and found significant performance disparities between them.

The accuracy of the systems tested often falls short of the claims of 99 percent or better accuracy, that are “based on idealized or simplified testing environments,” IPVM Research Engineer Dr. Mert Karakaya tells Biometric Update in an email.

More transparent, third-party validation is needed for vendor claims, he says, in particular to represent the complexities of real-world deployments that are not included in the design of NIST’s algorithm evaluations.

IPVM’s findings align with the observation from DHS’ 2022 Biometric Technology Rally that most errors occur on the image acquisition side of the facial recognition process, according to Karakaya. That makes the choice and placement of cameras critical to performance. Some systems showed strong performance with smaller watchlists, but struggled with a database of close to 2,500 faces.

Pixel density and face capturing were also issues for lower-performing systems.

“Vendors should consider IPVM’s facial recognition rankings as a call to action for more transparent and comprehensive testing that includes real-world variables, not just algorithmic efficiency,” Karakaya says. “By doing so, they can provide a more accurate and reliable indication of how their systems will perform post-deployment.

“For customers, these evaluations serve as a valuable resource for scrutinizing vendor claims and understanding the limitations of current facial recognition technologies,” he continues. “In terms of fitting in with other evaluations like NIST’s, this analysis adds a layer of real-world applicability that is often missing from more theoretical or laboratory-based assessments. While NIST provides an excellent baseline for algorithmic performance, combining those insights with practical, real-world testing will offer a more comprehensive view of a system’s efficacy.”

Article Topics

 |   |   |   |   | 

Latest Biometrics News


EU watchdog rules airport biometrics must be passenger-controlled to comply with GDPR

The use of facial recognition to streamline air passenger’s travel journeys only complies with Europe’s data protection regulations in certain…


NZ’s biometric code of practice could worsen privacy: Business group

New Zealand is working on creating a biometrics Code of Practice as the country introduces more facial recognition applications. A…


Demonstrating value, integrated payments among key digital ID building blocks

Estonia has achieved an enviable level of user-centricity with its national digital identity system through careful legislation and fostering collaboration…


Strata Identity launches uninterrupted identity services product

There are a few things that can be more annoying than your office computer logging you out of applications because…


Digital identities shaking up identity verification industry: Regula

The arrival of digital identities is shaking up how companies operate and verify identities. Regula has published a survey with…


Digital ID for air travel is DPI that could lift Africa’s economy, HID argues

As the African continent experiences economic growth and increasing global integration, the potential for enhancing air travel through improved digital…


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Most Read This Week

Featured Company

Biometrics Insight, Opinion

Digital ID In-Depth

Biometrics White Papers

Biometrics Events