Report indicates facial biometric accuracy improving but image quality could help further

A team of researchers from the Florida Institute of Technology and the University of Notre Dame have found generally higher false-match rates (FMR) but lower false non-match rates (FNMR) for images of African-Americans processed by facial biometric systems, but also that image quality may be a greater factor than the balance of quantity between different demographics.

For the report, “Characterizing the Variability in Face Recognition Accuracy Relative to Race,” (PDF) the researchers studied the results of two unnamed commercial off-the-shelf (COTS) matchers, and two convolutional neural network (CNN) -based matchers, VGG and ResNet. The results with the “COTS-A” and VGG matchers were consistent with previous studies that have found facial recognition is sometimes less accurate for matching women and people with dark skin. The newer “COTS-B” and ResNet matchers showed results even more accurate for African-Americans than for Caucasians.

The method or level at which accuracy differences are measured is also important, according to the report. The researchers considered ROC curve distribution, but found the method inappropriate for comparing face biometric accuracy across demographic groups with a fixed decision threshold. Considering differences at the level of imposter and genuine distributions, the researchers computed comparable d-prime for the ResNet face matcher distribution of both cohorts.

Image quality was found to be a significant issue, however, with 48 percent of images of African-Americans in the MORPH dataset found to be compliant with International Civil Aviation Organization (ICAO) standards, compared to 57 percent of images of Caucasians. Interestingly the VGGFace2 dataset used to train ResNet is estimated by the researchers to contain five or six images of Caucasians for every one of an African-American, indicating the matcher’s higher d-prime for the African-American cohort was not a product of demographic balance in the training dataset.

The researchers point out that SDKs for automatic ICAO-complaint image quality checks are available, and recommend more care be taken in image acquisition, making a comparison to procedures for iris biometric image capture.

IBM launched a new dataset specifically to help advance the industry’s understanding of diversity in facial biometrics earlier this year.

Article Topics

accuracy  |  biometrics  |  biometrics research  |  facial recognition