FB pixel

What information is stored in face biometric templates? EAB explores

Fraunhofer IGD expert discusses fairness of facial recognition systems


Deeply-learned face representations enable the success of current facial recognition systems today.

Despite the ability of facial recognition systems representations to encode the identity of an individual, however, recent works have shown that much information is stored within them.

A webinar organized by the European Association for Biometrics (EAB) and led by Philipp Terhörst, Research Scientist at Fraunhofer IGD, examined these issues on Tuesday.

The talk showed how many soft-biometric attributes are embedded in face biometric templates and that these attributes often have a strong correlation to face verification performance.

The event was divided into three parts, each answering a different question.

What information is stored in face templates?

According to Terhörst, the main information stored in face biometric templates include demographics, image characteristics, and social traits.

The research conducted by the Fraunhofer IGD scientist analyzed two face templates regarding 113 attributes, and concluded that 74 of them could easily be predicted, particularly non-permanent ones.

Information stored in Facenet (2015) templates- Credit Terhorst et all

Information stored in Facenet (2015) templates: Credit Terhörst et all

Terhörst achieved this by training a massive attribute classifier (MAC) to jointly predict multiple attributes, such as face shape, beard type, and whether the individual is wearing lipstick or not.

The biometric algorithms tested during the experiments were LFW and CelebA. Terhörst’s team analysed 13 thousand images from over five thousand individuals and up to 73 attribute annotations from LFW. The CelebA testing took into consideration 200 thousand images from over ten thousand celebrities and 40 binary attributes.

Information stored in Arcface (2019) templates Credit- Terhorst et all

Information stored in Arcface (2019) templates: Credit: Terhörst et all

The tests were carried out via FaceNet and ArcFace embeddings, and showed that head pose and social traits were the easiest to predict.

Face geometry, nose, and image quality, among others, were also predictable, while skin, mouth and environment were the hardest traits to predict.

How does it relate to fairness in facial recognition?

In order to assess these biases, Terhörst then proceeded to analyze the influence of soft-biometric attributes on the performance of facial recognition algorithms, particularly ArcFace and FaceNet.

To do so, the scientist’s team used a database named MAAD-Face holding a high number of face images, featuring several and high-quality attribute annotations.

These included six positive/negative control groups for each attribute, that were created by randomly selecting samples from the database. These synthetic groups had the same number of samples as their positive/negative counterparts.

“For example,” Terhörst explained, “if we had ten thousand sample images with eyeglasses, and ninety thousand without, for the positive control group we would just look for randomly selected ten thousand samples, and for the negative control group randomly selected ninety thousand.”

The results of analyzing both facial recognition algorithms through the control groups showed that, in terms of demographics, middle-aged, senior, white, and male individuals showed higher recognition accuracy rates than young, Asian, Black, and female individuals.

When visibility-related attributes were concerned, fully visible forehead, receding hairline, bald, non-eyeglass-wearing individuals scored higher accuracy rates than those with an obstructed forehead, bangs, eyeglasses and wavy hair.

Temporary attributes like hats, earrings, lipstick and eyeglasses also reduced the precision of the facial recognition algorithms, while arched eyebrows, big or pointy nose, bushy eyebrows, double chin and high cheekbones were responsible for higher accuracy rates.

Both algorithms also scored better with smiling faces and closed mouths, as opposed to individuals’ non-neutral expressions. According to Terhörst, this might be due to the fact that a vast number of images from the database were from smiling celebrities.

Additional biases regarded the colors of users’ hair and eye color, and whether they had a beard or not.

How can biases in facial recognition be mitigated?

Knowing encoded information in face templates might help to develop bias-mitigating solutions, Terhörst explained, proceeding to the third part of the webinar.

According to the scientist, however, previous works in this field required labels of the bias-related attributes beforehand, and could only mitigate specific biases.

These actions have also reportedly been known to degrade the overall performance of facial recognition algorithms, as well as present difficulties in integration into existing systems.

A possible alternative to traditional systems would be fair score normalization (FNS). The technique, Terhörst explained, can operate on unlabelled data, and effectively mitigate biases of unknown origin.

FNS can allegedly also improve the performance of facial recognition systems considerably, and can be integrated easily into existing systems.

How FNS works Credit Terhorst et all

How FNS works. Credit: Terhörst et all

Terhörst then concluded the event by taking questions from the audience.

The webinar was part of the EAB virtual events series on Demographic Fairness in Biometric Systems. Registration is still available for the events on March 15 and 30 to EAB members and non-members, with a discount for members.

Article Topics

 |   |   |   |   |   |   |   |   |   |   |   |   | 

Latest Biometrics News


Footprint makes a mark on ID verification market with $13M series A round

New York-based KYC and digital identity verification startup Footprint has announced a $13 million Series A funding round led by…


Pindrop claims up to 99% accuracy detecting synthetic voice fraud plaguing call centers

Pindrop has released its 2024 Voice Intelligence and Security Report, and its conclusions will come as no surprise to anyone…


Real-time remote biometrics banned in EU with final green light for AI Act

The European Union’s Artificial Intelligence Act received a final green light allowing it to become the world’s first major regulation…


Ethiopian capital Addis Ababa unveiled as host of 2025 ID4Africa AGM

It’s not only the case with sporting events like the FIFA World Cup, or the Olympic Games. The host of…


Ryanair accused of GDPR violations with biometric passenger verification

Travel policy advocacy group eu travel tech has lodged a formal complaint with the French and Belgian Data Protection Authorities…


Police in US cities that ban facial recognition asking others to do it for them

A major report in the Washington Post has found that law enforcement officers in U.S. several cities where facial recognition…


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Most Read This Week

Featured Company

Biometrics Insight, Opinion

Digital ID In-Depth

Biometrics White Papers

Biometrics Events