John Howard reflects on DHS biometrics evaluations and the AI testing landscape

When John Howard began working for SAIC at the Maryland Test Facility, biometrics was an emerging technology supported by a nascent industry. When he announced his departure near the end of January, the MdTF and Howard personally were long-established as central figures in proving the science and setting the standards behind facial recognition and other biometrics.

Howard’s lengthy list of contributions as senior principal scientist with SAIC ranges from helping set up and scale assessments of operational biometrics deployments in projects like the DHS Biometric Technology Rally, RIVTD and RIVR, to research and standard development around demographic fairness.

Reflecting on highlights from his time with MdTF in an email interview with Biometric Update, Howard says, “Throughout my career, one of my biggest areas of research has been into demographic effects (‘bias’) and how to test complex AI systems in operations.”

He points out that a system that always fails for every user group could be considered equitable, so usefulness is a function of both overall accuracy and fairness.

“Looking ahead, I believe this work will lay the foundation for how we — as a society and as individual companies – reason about 1) the operational realities of AI systems, 2) deep fakes, and 3) effective AI and data policies.”

Howard’s next project is Sensus AI, the independent testing, assessment, audit and advisory firm he co-founded and will lead as CEO.

Howard makes clear that his departure from MdTF does not reflect any diminished role for the lab in facial recognition policy and development.

“The U.S. Government has a significant role in guiding the future of facial recognition technologies, and government laboratories are going to be critical in providing expertise on the development, evaluation, and deployment of those technologies in the government space,” he says.

Evaluating evaluations

One of Howard’s last projects with MdTF was a paper on face biometrics image quality evaluation tool OFIQ. It suggested that the open-source tool is very limited in its usefulness to DHS, and its quality filter “did not substantially reduce error rates.”

Christoph Busch of Hochschule Darmstadt pushed back on those findings somewhat during an OFIQ user group meeting, suggesting that the criteria of the assessment is not as applicable to other users, like eu-LISA.

Howard does not entirely agree, but sees the debate as an example of the need to apply appropriate expertise to get real-world applications right.

“Airports and other large venues are going to face challenges as they introduce face recognition programs because they are complex environments,” he points out. “For most high-throughput deployments of face recognition systems, you don’t want to discard usable face samples. Our paper showed that is what OFIQ does. Every implementation has its subtleties, eu-LISA included, and that highlights the importance of expert consultation when operationalizing these technologies.”

Consequences of success

When asked about how helpful the biometrics testing and evaluation ecosystem is when it comes to evaluating vendor claims and the applicability of technology to specific use cases, Howard says that it “could do better.”

“Right now, there’s no shortage of people eager to build and deploy AI across all kinds of applications. What is missing, especially in the private sector, is a sufficient number of professionals focused on independent AI test, evaluation and auding. I believe that in the next four years, every organization is going to need independent, qualified, AI test, evaluation — much like organizations do financial audits today. That’s not the model we are currently operating under but I hope we move in that direction.”

As AI and biometrics adoption is followed by enhanced scrutiny and then governance, Howard also sees a need for realistic transparency.

“The industry should be clear to the public that these systems make mistakes,” he writes. “They are constantly getting better and honestly the progress has been remarkable but that success can sometimes lead people to stretch these systems into ever more applications. A system that makes very few errors in one use case can produce an astronomical number of mistakes in another.”

Article Topics

biometric bias  |  biometrics  |  biometrics research  |  demographic fairness  |  DHS S&T  |  facial recognition  |  Maryland Test Facility (MdTF)  |  Sensus AI