Hire the right people to be in the facial recognition loop to dramatically raise accuracy

Choosing the right humans to be in the loop of facial recognition decisions, and utilizing them in the right workflow can dramatically improve the accuracy delivered by biometric algorithms alone, studies conducted in Australia show.

A webinar on ‘Facial recognition in identity management: AI and humans working together for a secure Australian passport’ was presented by a University of New South Wales researcher and hosted by the UNSW Institute for Cyber Security.

David White studies how people perceive faces, and has worked with the Australian Passport Office and NIST among other organizations.

Australia was already issuing 12,000 passports per day before the Optus hack raised the possibility that millions of passports will have to be reissued. The biometric chips embedded in passports compliant with the latest ICAO standards means that it is impossible for the hackers to produce fraudulent passports from the stolen data, but passport fraud of various kinds is still possible.

White reviewed how fraudulent passports can be obtained and how facial recognition can help prevent their issuance.

A fraudster could take personal information from among hacked data to determine that an individual has an expired passport but is not likely to apply for another, perhaps because they have died or been incarcerated. The data could be used with the photo of the fraudster to apply for a passport that would then allow the criminal to assume someone else’s identity. Identities that have never applied for a passport before can also be used in similar fraudulent processes.

A facial recognition search can reveal that the fraudster already has a passport, but it is a passport review officer who must make the determination, based on the software’s input, to pass the application to a forensic review process.

White notes that NIST research has found that matches with a significant number of images will be indistinguishable from false positives, introducing the need for human review.

Like algorithms, not all human review equal

“Now this is a problem,” the cognitive psychology expert explains, “because humans aren’t particularly good at matching unfamiliar faces.”

White showed a pair of facial images and asked the audience if they are the same person. They are, and taken only minutes apart in similar lighting conditions, but the use of different cameras changes the aspect ratio, making one face appear slightly wider.

Familiar faces, even in images with wider variance, pose much less of a challenge to most people.

In collaboration with the passport office, UNSW tested staff’s ability to match faces, and found passport officers, like university students, are wrong about 20 percent of the time. When asked if an applicant face matches a potential imposter drawn from a gallery of faces with high similarity scores, the error rate of the human reviewers jumped to half.

Identity verification with facial recognition should also make use of human review, White says, at least for some image pairs, despite major advances in biometric accuracy over the past five years. White was part of a study that identified high and low certainty thresholds between which human adjudication can improve accuracy, and in the most recent test, about 2 to 2.5 percent of checks fell within this threshold.

UNSW researchers including White collaborated on a study with NIST that suggested the general conclusion back in 2018.

White and others working with the Australian passport office have proposed solutions to these challenges based on human resources and workflow design.

Human performance in fac matching is highly variable between individuals, as a large body of research now shows, and the differences appear to be largely inherited, rather than a result of training.

Choosing the people with the most face matching talent, then, and using their ability in fusion with biometric algorithms, will yield the best results, according to White. The wisdom of crowds can also be utilized by aggregating responses, for accuracy gains as high as 20 percent.

One of the reasons the fusion approach works so well, White explains, is that algorithms and humans are each capable of rejecting possible pairs that the other would accept. They rank different pairs of faces from a given group as the least-similar.

White also touched on future challenges, including face morphing.

Article Topics

accuracy  |  biometric passport  |  biometrics  |  biometrics research  |  face biometrics  |  facial recognition