What is emotion recognition, and how can biometrics be used to perform it?

At a basic level, emotion recognition (ER) is the process of identifying human sentiment. While there have been many contributions to this discipline from a psychological and philosophical standpoint, the same cannot be said about technology. Attempting to identify human emotion using technology is a relatively new field and one that has been met with mixed results.

Within the realm of these applications, however, biometric approaches have shown promise, particularly face biometrics. The combination of different physiological and behavioral data types, and use of multimodal biometrics have also helped the discipline move forward.

But how can biometric scanners and algorithms measure human emotion in practical terms?

Face biometrics

Most ER tools based on face biometrics work via readings of physiological responses and facial expressions. The latter are categorized into discrete emotions and typically include amusement, awe, enthusiasm, liking, surprise, anger, disgust, fear, and sadness.

As described in a 2018 study by Universiti Putra Malaysia, facial emotion recognition (FER) processes are usually divided into three stages, face detection, feature extraction (nose, mouth, etc.), and emotion classification.

For a FER system to work, in the last stage of this process, a classifier needs to be trained in order to generate labels for individual emotions using the training data. Different facial coding systems have been developed to classify emotions based on the movements of specific facial muscles.

Voice recognition

ER based on voice biometrics is still in its infancy. Still, as voice recognition algorithms become increasingly more efficient, researchers and companies alike are working to apply the technology to measuring sentiment.

Technically, speech ER (SER) can rely on three distinct characteristics of the human voice, namely lexical features (vocabulary), visual features (facial expressions) and acoustic features (sound properties).

According to a 2020 study by Brillio, SER systems relying on lexical features require a transcript of the speech alongside text extraction capabilities to predict emotions from real-time audio. Those depending on visual elements, on the other hand, rely on the analysis of the video of the conversations, making it a form of multimodal biometric ER.

Because of these limitations, many SER rely on acoustic features, which can be analyzed in real-time. The Brillio studio mentioned here, in particular, tried to classify emotions based on two methods.

The first one, called discrete classification, is akin to what is mentioned above for FER, categorizing emotions in discrete labels like anger, happiness, boredom, etc. The second one, “dimensional representation,” aims to represent emotions with dimensions such as Valence (on a negative to positive scale), Activation or Energy (on a low to high ranking) and Dominance (on an active to passive scale).

It is helpful to mention here that with both face and voice biometrics ER tools (as well as standalone devices), wearables are often used to record physiological data, including cardiovascular information, to identify emotions more accurately.

Gait recognition

Methods of reading individuals’ emotions from their gait biometrics are slowly improving, but the technology is still relatively far away from commercial applications.

Still, there have been several attempts by researchers and governments to turn gait ER (GER) into a viable method of assessing a person’s emotional status.

According to a non-peer-reviewed study published by Chinese researchers in September 2022, for instance, it is possible to assess an individual’s emotion based on the position of certain bones in their skeleton as they walk.

The algorithm used in this study, called multi-scale adaptive graph convolution network (MSA-GCN), is intended to extract general gait data and create fine-grained graphs to uncover local information.

A similar technique (generally understood as the most widespread technique to perform GER at the time of writing) was presented in a separate paper published in January 2022 by a joint team of Canadian and Taiwanese researchers.

Legal implications

Biometric surveillance is generally among the most criticized methods of technocratic control by authoritarian regimes worldwide, and is often associated with emotion recognition. ER to many people evokes dystopian images of mind-controlled citizens, immediately punished for thinking or feeling something they should not.

In China, for instance, the government has deployed facial recognition tools to record students’ moods or ones allegedly capable of testing loyalty to the Party.

Beyond the alleged and actual Orwellian schemes associated with ER deployments, these technologies are finding fertile ground among lawyers in the U.S., who have been reportedly using them to assess the emotions of panels hired by attorneys before trials as stand-ins for juries.

Meanwhile, UK politicians are treading carefully regarding emotion recognition technologies. The country’s Information Commissioner’s Office (ICO) issued a warning against the deployment of these technologies in October 2022.

Article Topics

biometrics  |  emotion recognition  |  face biometrics  |  gait recognition  |  voice biometrics