IPVM launches evaluation of facial recognition systems in real-world conditions

Security camera trade publication IPVM has published its first-ever Facial Recognition Rankings, which indicate significant differences between the performance claimed by vendors and the accuracy of their commercial systems in real-world conditions.

The evaluation included tests of the various commercial systems in a range of scenarios. Each vendor was assessed for performance on production at different angles of camera tilt and subject approach, with subjects wearing accessories, in groups, and captured with low, normal, and infrared lighting. Rankings are provided in each category, along with information on usability and installation. IPVM plans to expand the evaluation in future editions.

Biometric enrollment and identification were assessed using the NIST IJB-C dataset.

The surveillance camera compared facial recognition performance by commercially available cameras from Avigilon, BriefCam, Hikvision, IronYun, Oosto, Rhombus, Turing and Verkada, and found significant performance disparities between them.

The accuracy of the systems tested often falls short of the claims of 99 percent or better accuracy, that are “based on idealized or simplified testing environments,” IPVM Research Engineer Dr. Mert Karakaya tells Biometric Update in an email.

More transparent, third-party validation is needed for vendor claims, he says, in particular to represent the complexities of real-world deployments that are not included in the design of NIST’s algorithm evaluations.

IPVM’s findings align with the observation from DHS’ 2022 Biometric Technology Rally that most errors occur on the image acquisition side of the facial recognition process, according to Karakaya. That makes the choice and placement of cameras critical to performance. Some systems showed strong performance with smaller watchlists, but struggled with a database of close to 2,500 faces.

Pixel density and face capturing were also issues for lower-performing systems.

“Vendors should consider IPVM’s facial recognition rankings as a call to action for more transparent and comprehensive testing that includes real-world variables, not just algorithmic efficiency,” Karakaya says. “By doing so, they can provide a more accurate and reliable indication of how their systems will perform post-deployment.

“For customers, these evaluations serve as a valuable resource for scrutinizing vendor claims and understanding the limitations of current facial recognition technologies,” he continues. “In terms of fitting in with other evaluations like NIST’s, this analysis adds a layer of real-world applicability that is often missing from more theoretical or laboratory-based assessments. While NIST provides an excellent baseline for algorithmic performance, combining those insights with practical, real-world testing will offer a more comprehensive view of a system’s efficacy.”

Article Topics

biometric data quality  |  biometric testing  |  biometrics  |  facial recognition  |  IPVM  |  video surveillance