Programmer builds a better lie detector with computer vision and facial analysis

A remote lie detector based on biometric signals has been developed by Fletcher Heisler, host of the ‘Everything is Hacked’ YouTube channel. The Truthsayer software uses standard video feeds to analyze changes in the expression of subjects to find indications that they are not telling the truth.

Hackaday covered the software and a highly engaging presentation Heisler delivered on it at DEF CON 30, which has been posted to YouTube.

Heisler drew attention in the computer vision community earlier this year when he created software for coding with facial expressions.

Heisler notes the limitations of polygraph tests. Their effectiveness is “controversial at best,” they cannot be performed remotely, and the subject always knows they are taking one.

He used OpenCV, a pre-trained mood recognition library, and Face mesh from Google’s MediaPipe to build Truthsayer in the Python programming language.

Detecting the face of the subject, particularly with footage including multiple people, was a temporary challenge in Truthsayer’s development, Heisler says.

Heart rate is measured from the rush of blood to and from the face as picked up by a standard webcam. Changes are measured, along with differences in blink rate, average gaze direction, lip compression and occlusions of the face with the subject’s hand. Clusters of signals can indicate dishonesty.

“There’s a lot of pseudoscience here,” Heisler acknowledges, “but there’s a lot of really well-researched science.”

The result is a slide-bar with truth on the right, and considered as the baseline, and ‘deceit’ on the right.

U.S. Congressional testimony from Mark Zuckerberg showed no signs of deception, but Truthsayer identified moments in his testimony to European Parliament where questions about shadow profiles prompted signs of deceit.

Potential uses of the software suggested by Heisler include social engineering, but also increasing trust in negotiations and interviews or online dating, assisting therapy or guided meditation, and as an aid for autistic people to understand the expressions of others.

Article Topics

biometrics  |  computer vision  |  Def Con  |  expression recognition  |  face biometrics  |  facial analysis  |  lie detection  |  research and development  |  video analytics