New skin tone evaluation scale developed to improve AI fairness evaluation

An innovative development in AI skin tone measurement methods, aimed at refining how artificial intelligence interprets skin tones across diverse populations, has been introduced to help address previous biases in digital applications. The research is presented in an October 2024 paper hosted on arXiv, which introduces a new Colorimetric Skin Tone (CST) for accurately measuring skin tone, and announced in a LinkedIn post by John Howard, principal scientist at The U.S. government’s Maryland Test Facility (MdTF).

The growing demand for more inclusive AI-driven technologies has spurred the development of new evaluation criteria and methodologies. Historically, AI systems have struggled with accurately representing skin tones, often resulting in biases that predominantly affect people with darker skin. Howard emphasizes the importance of skin tone evaluation as a necessary step toward fostering AI inclusivity.

The latest research, detailed in the arXiv paper, expands on the Fitzpatrick skin type scale, traditionally used to classify skin tones based on responses to UV exposure, to include more nuanced variations in tone. The researchers, led by Isabelle Shuggi of SAIC’s Identity and Data Sciences Laboratory, measured the relationship between the classifications from the Fitzpatrick and Monk Skin Tone (MST) scales and the readings of a calibrated colorimeter.

Colorimetric measurements were used to generate the CST, which the researchers say “is more sensitive, consistent, and colorimetrically accurate” than manual classification by humans.

By introducing this broader classification system, researchers aim to provide a more granular approach to skin tone representation. According to the study, existing skin tone measurement systems lack precision and inclusivity, potentially causing AI systems to misidentify or inadequately represent individuals from diverse backgrounds.

The research outlines how training machine learning models on datasets compiled using the new skin tone metrics can combat these biases.

Howard’s discussion and the arXiv publication both emphasize the impact of inclusive AI on societal fairness, particularly in fields reliant on accurate skin tone representation. It also builds on Google’s 2022 release of an expanded skin tone scale, a tool developed to mitigate bias in AI and computer vision.

The researchers also found several complicating factors for classifying skin tone, such as different judgements by people based on the background a given subject is presented against.

“Although human annotation of skin tone may never be objective,” the researchers conclude, “the process can be improved by using the CST scale.”

Article Topics

biometric research  |  biometrics  |  Colorimetric Skin Tone (CST)  |  demographic fairness  |  Maryland Test Facility (MdTF)  |  skin tone scale