Limitations of FRT apparent in search for United Healthcare CEO’s killer

The murder of United Healthcare CEO Brian Thompson in Midtown Manhattan involved the use of facial recognition technology (FRT) to identify his killer, but challenges hindered its success. Partial images of the suspect captured on some of the more than 15,000 various advanced surveillance systems and security cameras (including from inside taxis and drones) throughout New York City (NYPD) failed to capture the suspect’s full facial features, limiting the effectiveness of the technology in isolating the suspect’s identity.

After five days and no success in identifying their suspect using FRT, the NYPD released yet more photos it said were the clearest it had of the suspect to solicit information from the public​.

It was those photos that led to a public tip that the suspect appeared to be the same man who was being observed in a McDonald’s in Altoona, Pennsylvania about 200 miles from New York City. That man, identified as Luigi Mangione, was promptly detained by local police and eventually charged with Thompson’s murder.

“That photo has been seen more times than in your average homicide,” former Philadelphia police officer and a criminal justice professor at The Citadel Sean Patrick Griffin told The New York Times Monday.

The Times added that, “In the end, it was the simple act of distributing photos – not sophisticated facial recognition technology – that led the police to the man who has been charged in the fatal shooting … For experts, the case was a reminder of how – even as facial recognition technology grows more sophisticated – distributing photos and relying on the public to recognize a face can still play a critical role in investigations.”

Futurist magazine’s The Byte said: “AI Completely Failed to Catch CEO Killer.”

But even if the NYPD had had a clear facial image of the suspect, it would only have been useful if the suspect’s face was in its criminal database, the New York Department of Motor Vehicles’ biometric database, the Federal Bureau of Investigation’s (FBI) Next Generation Identification (NGI) database, the Department of Homeland Security’s Automated Biometric Identification System (IDENT), or the Department of Defense’s Defense Biometric Identification System (DBIDS).

While facial recognition technology has been widely employed by law enforcement to identify suspects, its role in identifying Mangione was ineffective. But perhaps not solely because of the technology’ shortcomings. Legal and privacy issues and federal and interstate database connectivity restrictions also may have played a role.

Officially, the NYPD says its primary facial recognition tool is provided by DataWorks Plus. However, the department acknowledges the use of other tools for specific purposes and has faced criticism for a lack of transparency and comprehensive oversight regarding AI technologies​.

DataWorks’ FaceCompare Plus provides morphological facial comparison of any two facial images side-by-side, without the need for a full facial recognition system. But, again, Mangione’s photo or photos would have to have already been in the NYPD’s or other state and federal databases to be useful.

“Most Americans may believe that law enforcement has images on everybody in the United States. That’s very much not true,” Idemia Public Security CEO Donnie Scott was quoted by CNN. “If he happens to not be a resident of New York who happens to not have been arrested before, odds are he’s not going to be in their criminal database or their mugshot repository.”

Scott further emphasized that “the state of New York does not have access to the DMV database for law enforcement purposes by statute. It requires cooperation and information sharing and a reason and willingness by the respective agencies to be allowed to share that by law.”

Similarly, NYPD – and many other law enforcement agencies nationwide – do not have immediate access to one another’s databases, least of all their state’s DMV database. Neither does one state have access to another state’s DMV photo repository.

The Federal Bureau of Investigation (FBI) does, and it confirmed its involvement in the investigation of Thompson’s murder as it joined efforts with the NYPD after the incident, which took place outside a Manhattan hotel. The FBI assisted in the manhunt for the suspect by leveraging its resources to track leads and identify the perpetrator. It’s not known if the FBI’s efforts resulted in some database photo match.

The FBI has access to state law enforcement criminal databases and DMV records, but this access is regulated and governed by federal and state laws, often through cooperative agreements.

The FBI operates the Criminal Justice Information Services (CJIS) Network which includes systems like the National Crime Information Center (NCIC). State and local law enforcement agencies input and access data on the NCIC, and the FBI can use this data for its investigations.

The CJIS network also connects state and local databases to the FBI, enabling data-sharing across jurisdictions​.

As for DMV Records, the FBI can access these records for investigations, often through the Driver’s Privacy Protection Act, which permits such access for legitimate law enforcement purposes. Through the National Law Enforcement Telecommunications System, the FBI can request state-level DMV data, which includes driver license information and vehicle registrations.​

Access though often depends on agreements between state law enforcement agencies and the FBI. States maintain their own databases but grant access to the FBI when investigations require it, under established protocols. Access is also closely monitored to ensure compliance with privacy laws, and misuse of these systems can result in significant penalties.

In the end, however, it was partial photos of the suspect that the NYPD provided to the public through the media that resulted in Mangione’s arrest – that, after the NYPD had made a big deal about its investigation.

Former NYPD Commissioner Ray Kelly told FOX Business early on that he believed the suspect would be caught “fairly soon” and that “facial recognition is effective, and I would hope that it’s being used in this case.”

NYPD Chief of Detectives Joseph Kenny added early on in the investigation, however, that police so far hadn’t been able to ID him using facial recognition, possibly because of the partial images or limitations on how the NYPD is allowed to use the technology.

While facial recognition has evolved significantly, modern systems still have difficulty handling all the challenges like surgical masks, balaclavas, or partial images, which have varying degrees of success.

Facial recognition systems still face significant challenges when analyzing partial images that are obstructed by masks or balaclavas. While advanced systems can focus on visible features like eyes, eyebrows, and facial contours, accuracy diminishes when large parts of the face are hidden. Performance depends on factors like resolution, image quality, and the algorithms used.

Some systems trained for such scenarios have improved recognition rates, leveraging AI models that extract additional clues from limited data. However, the success rate often remains significantly lower compared to unobstructed images. Studies that were conducted during the COVID-19 pandemic showed accuracy drops ranging from 5% to 50%, depending on the system and the coverage of the face.

Many systems have been updated to accommodate masked faces by focusing on visible features like the eyes, eyebrows, and forehead, and some companies, like NEC and Huawei, have claimed that their technologies achieve 90% -95% accuracy even with masks.

Advanced systems using 3D mapping and infrared imaging can bypass some of the limitations of traditional 2D recognition, and some AI models that have been trained with extensive datasets, including masked faces, perform better than those trained on unmasked datasets.

Feature-based matching uses algorithms to analyze parts of the face that are visible (e.g., eyes, forehead) and compare them with a database, relying heavily on unique features like the shape of the eyes or eyebrows. However, the fewer visible features, the lower the accuracy. Most systems see a significant drop in performance when more than 30% to 50% of the face is obscured.

FRT systems also frequently struggle with extreme angles or occlusions, as partial images introduce uncertainty. Advanced techniques like pose normalization or multi-shot analysis (combining multiple partial images) can help mitigate these issues.

AI models, particularly convolutional neural networks (CNNs), have shown considerable promise in identifying individuals with partially obscured faces. These models are designed to focus on specific facial features that remain visible, such as the eyes, eyebrows, and forehead. They can also use contextual clues and patterns from datasets trained on partially masked faces.

However, their success rate depends on the extent of obstruction, quality of the image, and robustness of the training dataset. State-of-the-art models achieve higher accuracy but still face challenges under poor lighting, movement, extreme angles, or significant obstructions.

Generative Adversarial Networks (GANs) can be highly effective in assisting facial recognition for partially obscured faces by reconstructing or inferring missing portions of a face by generating plausible approximations based on training data. This reconstruction allows other recognition systems to analyze a complete or near-complete representation of the face.

However, while GAN-enhanced systems can improve identification rates, their accuracy depends on the quality of the training data and the degree of obstruction of a person’s face. GANs also face risks of introducing artifacts or false positives if misused.

The exact number of facial recognition systems using CNNs in law enforcement though is difficult to pinpoint due to their proprietary nature and varied implementations. However, major providers like Clearview AI, NEC NeoFace, Amazon Rekognition, and Cognitec employ CNN-based algorithms in systems widely adopted by law enforcement agencies globally.

While Clearview AI, for instance, uses a CNN to analyze over 50 billion imagesv from public sources, even with partial or low-quality images, it isn’t known whether any of the numerous photos of Mangione found on social media and other online public sources after he was identified are included in its database, or whether its CNN would have been able to identify him as a possible match.

Overall, while FRT has made strides in addressing its many challenges, performance can vary widely depending on the specific system and context.

Meanwhile, discussions about facial recognition and data-sharing frameworks continue in the broader context of law enforcement technology and privacy debates, with calls for stronger regulation and training to address privacy and civil rights concerns.

Efforts to legislate or regulate facial recognition in the U.S. remain contentious. Some lawmakers and advocacy groups push for stricter limits or bans on certain uses, citing privacy concerns and the potential for misuse. On the other hand, others advocate for better integration and use of this technology for national security and crime prevention, though this, too, is often met with privacy-focused resistance.

In the wake of Thompson’s murder and the inability of FRT to identify Mangione, expect this debate to only grow louder – on both sides. Several federal law enforcement officials expressed in private they are worried that there will be copycats, especially given the surprising level of galvanized public support from the right and the left – elites versus the little guy – for what Mangione did. “This reverse outrage,” as one official described it, “is truly troubling. I would expect some sort of political response that tears down restrictions” on the use of facial recognition “going forward.”

Article Topics

biometric identification  |  biometrics  |  data privacy  |  data sharing  |  DataWorks Plus  |  facial recognition  |  Idemia Public Security  |  legislation  |  New York City  |  police  |  regulation