New NIST FRTE 1:N results illustrate facial recognition accuracy’s multidimensionality

NIST released its latest Face Recognition Technology Evaluation (FRTE) 1:N results on September 4, adding submissions from seven new developers and eight returning developers to its ongoing evaluation of facial identification technology.

The new developers are Innov8tif Solutions, Integrated Software and Technologies Limited, Intpixel Labs, Thakaa Advanced Company for Information Technology, TrueSight Laboratories Tech, UniverseAI and Virtisha AI. Returning developers include Iris ID, Miaxis Biometrics, Nominder, Panasonic, ROC, SenseTime, Shanghai Jiao Tong University and Vsoft Tecnologia.

TrueSight Laboratories made a strong debut, reaching the upper end of several rankings. Panasonic’s latest algorithm showed gains across multiple scenarios, while SenseTime’s new submission posted leading technical results across several tests. Established providers including NEC, Idemia and Paravision also remain near the front of the field.

One point is important when reading the numbers. NIST reports identification accuracy as false negative identification rate (FNIR) at a threshold set to limit false positive identification rate (FPIR) to 0.003. The threshold is set separately for each algorithm and test, meaning the FNIR figures below are compared at the same target false-positive operating point rather than allowing an algorithm to lower its miss rate simply by accepting more false positives.

TrueSight makes a strong entrance

Among the new developers, the mysterious TrueSight Laboratories Tech stands out for the breadth of its first results.

The impressive debut follows a similarly strong result in NIST’s August 1:1 FRTE update. The company is registered as an LLC in the United States, but does not appear to have a website at this time.

TSIGHT-000 reached the top five across several identification scenarios, ranking fifth in mugshot-to-mugshot, fourth in mugshot-to-webcam, fourth in mugshot-to-profile and fifth in border-to-border searches. It also placed second in visa-to-kiosk and tenth in visa-to-border testing.

Its performance across NIST’s smaller gallery sizes is particularly notable. Against approximately 640,000 identities, TSIGHT-000 records an FNIR of 0.05 percent and ranks second. At 1.6 million identities it remains second at 0.05 percent. At three million identities, it ranks third while FNIR increases only slightly to 0.06 percent.

The gallery grows almost fivefold while the error rate barely changes.

The picture changes at NIST’s larger gallery sizes, however, where the submission does not maintain that performance. The contrast illustrates why gallery size matters when interpreting an identification accuracy figure.

TrueSight’s broader showing nevertheless makes its first appearance notable, with competitive results across mugshot, webcam, profile, border and kiosk imagery.

Panasonic shows progress across multiple scenarios

Panasonic tells a different story, with its latest submission showing progress across several tests.

Panasonic’s v6 algorithm enters the top ten in three scenarios. It ranks ninth in mugshot-to-mugshot identification with an FNIR of 0.2 percent, fifth in visa-to-border at 0.1 percent and tenth in border-to-border testing at 1.9 percent.

Panasonic also remains among the leading algorithms in other scenarios through its earlier submission, including mugshot-to-profile and visa-to-kiosk identification.

The movement is noteworthy because it spans multiple capture environments rather than being confined to one dataset.

SenseTime combines accuracy with consistency

SenseTime’s latest submission is another notable technical development.

SenseTime v12 ranks third in mugshot-to-mugshot identification with an FNIR of approximately 0.1 percent and fifth in mugshot-to-webcam at 0.7 percent. It also reaches fourth in visa-to-border and tenth in mugshot-to-profile testing.

Its gallery-size results provide another perspective. SenseTime-012 records an FNIR of approximately 0.06 percent against 640,000 identities, 0.07 percent against 1.6 million and 0.07 percent against three million. At six million identities, FNIR remains around 0.14 percent before rising to approximately 1.95 percent at 12 million.

The results show strong technical performance, but SenseTime also carries significant regulatory and human-rights considerations for potential buyers. The company has been on the U.S. Commerce Department’s Entity List since 2019. In 2021, the U.S. Treasury also identified SenseTime as a Non-SDN Chinese Military-Industrial Complex Company, citing facial recognition technology developed by its Shenzhen subsidiary that Treasury said could determine ethnicity, with a particular focus on identifying Uyghurs.

For governments and enterprises evaluating biometric suppliers, the distinction is important: NIST measures algorithm performance, not the broader regulatory, geopolitical, human-rights or procurement considerations associated with a vendor.

Different problems produce different leaders

Leadership changes across NIST’s different test scenarios.

Idemia leads the mugshot-to-mugshot and mugshot-to-webcam rankings examined in the latest release. NEC leads border-to-border and visa-to-kiosk identification. Paravision takes the top position in visa-to-border, while Fujitsu leads mugshot-to-profile.

The differences reflect the underlying identification problems. Matching relatively controlled mugshots differs from identifying someone captured through a webcam, while profile images, kiosks and border environments introduce their own pose, camera, lighting and capture conditions.

For buyers, the relevant benchmark therefore depends on the environment in which the technology will be deployed.

Gallery size adds another dimension

NIST’s results also highlight a less-discussed characteristic of 1:N facial recognition: how performance changes as the gallery grows.

Across algorithms in the analyzed data with results at all five gallery sizes, median FNIR rises from 0.61 percent at 640,000 identities to 1.24 percent at three million, 2.58 percent at six million and 10.34 percent at 12 million. Algorithms that look similar against smaller galleries can therefore separate considerably at larger scales.

But maintaining a database of millions of identities does not mean every transaction needs to search all of them. Large systems can use contextual information to create smaller candidate galleries. CBP’s Traveler Verification Service, for example, uses advanced traveler information to assemble galleries of expected travelers rather than searching its entire biometric repository for every passenger.

The appropriate approach depends on the use case. Some applications can substantially narrow the search population, while national identity deduplication or other open-set searches may offer fewer opportunities to do so.

For governments and large enterprises, scalability therefore becomes both an algorithm and system-design consideration: how large a gallery does the use case require, how effectively can it be optimized, and can the algorithm maintain the required accuracy and speed at that scale?

From benchmark to deployment

The September results show movement on several fronts: a strong first submission from TrueSight, broader gains from Panasonic and a competitive new algorithm from SenseTime, alongside continued strong results from established providers.

For buyers, NIST provides an important technical benchmark, but deployment decisions extend beyond the leaderboard. Capture conditions, gallery size, demographic performance, speed and system architecture all need to be matched to the intended application. Vendor-level regulatory and procurement considerations can add another layer.

The benchmark tells organizations how an algorithm performs under NIST’s test conditions. The next question is how those results translate into the system they actually need to operate.

Article Topics

accuracy  |  algorithms  |  biometric identification  |  biometrics  |  Face Recognition Technology Evaluation (FRTE)  |  facial recognition  |  NIST