Test lab, bias considerations for Australia age assurance trial revealed

Testing and evaluation of age assurance technologies for the upcoming trial in Australia will be carried out by Melbourne-based software consultancy KJR, InnovationAus.com reports. One of the early challenges identified is in ensuring biometric age estimation and other technologies work as well for Australia’s First Nations people as they do for everyone else.

The trial is being organized by the UK-based Age Check Certification Scheme (ACCS), and has drawn participation from around 50 age assurance providers.

Among them are familiar names in age assurance like Yoti, Incode, Civic, Fujitsu, IDVerse, iProov, Persona, Trust Stamp and VerifyMy. iProov, which provides face biometrics and liveness detection for the Australian government’s MyGov app, is there, as is domestic developer Blue Biometrics and French developer Needemand.

The testing will provide assurance and help keep stakeholders honest, a representative of KJR says. This is important in part because the effectiveness of some of the technologies, like Needemand’s, are so new they have not gone through independent testing.

“All the indications are yes, it’s mature,” KJR General Manager Andrew Hammond told InnovationAus.com. “And certainly, industry has changed its view from a few years ago, where it was that this was never going to be possible to now it is possible.” More testing is necessary, however, to give a conclusive answer to how reliable the age assurance industry’s boldest claims are, he says.

Then there are concerns about bias challenges that are universal in type, but specific to the demographics of the Australian market. Following that set of tests, a wider trial will test the technology with people between 13 and 23 years old from various demographic groups, including First Nations people, in school settings.

Hammond notes that models developed in the U.S. or UK “are unlikely to have our First Nations people in there,” so testing the systems for bias caused by underrepresentation in training data will be necessary.

As biometrics testing lab Fime notes, “There is now consensus that an AI model is only as good as the data that trained it.”

Improving biometric bias testing

Biometric bias testing has evolved significantly in recent years, as described in a recent blog post from Fime.

The primary parameters for accuracy in a biometric system are false accept rate (FAR) and false reject rate (FRR), which form the basis for measuring disparities. Those measurements take the form of metrics like “Fairness Discrepancy Rate (FDR), Inequality Rate (IR), Gini Aggregation Rate for Biometric Equitability (GARBE) or Separation Fairness Index (SFI).”

Fime details the method for a test to evaluate the sensitivity of the metrics.

“Once the fairness metrics on the data sets for both the unbiased and the synthetically biased scenarios had been computed, using the Pearson Correlation Coefficient allowed the experts to visualize the linear relationships between the metrics and the bias introduced,” the post explains. “They could then compare how the fairness metrics responded to each of the synthetic alterations. Metrics controlled by an alpha parameter – a variable value used in FDR, IR, and GARBE to achieve equity between security and the user experience – are less stable than those without one.”

Ultimately, Fime proposes the use of Area Max Differential Rate (AMDR), which measures the differential between false match rate (FMR) and false non-match rate (FNMR), as an improved fairness index.

Article Topics

Age Check Certification Scheme (ACCS)  |  age estimation  |  age verification  |  Australia  |  biometric testing  |  biometric-bias  |  biometrics  |  Fime