FB pixel

SenseTime researcher presents progress on AI syncing faces with audio

Categories Biometric R&D  |  Biometrics News
SenseTime researcher presents progress on AI syncing faces with audio

The digital manipulation of faces can be used to spoof biometric systems or sow misinformation, but technology to coordinate face movements with the speaker’s voice are also in demand in several areas, as SenseTime Researcher Yuxin Wang explained during a talk for the European Association for Biometrics (EAB).

His presentation on ‘Talking Faces: Audio-to-Video Face Generation’ was part of the EAB’s Workshop on Digital Face Manipulation & Detection, held this week for members of the organization.

Digital technologies have been used since the 1990s to generate synthetic video of people talking, for applications like virtual assistants, teleconferencing, movie and video game dubbing and digital twins.

The output in talking head generation should have “much more head movement” than the source material in audio-driven face re-enactment, Wang says.

Wang reviewed the modelling techniques that allow for relationships between head movement and vocalization to be measured for talking face generation. Data pulled from an audio representation is used to make the mouth movement and expressions of the speaker in the video accurately and consistently match the sound.

He described a pair of pipelines to do so; one based on audio and image encoders producing representations run through a single decoder, and the other using a regression model on audio features for combination with an intermediate feature, like a facial landmark, and rendered from the intermediate feature. Wang also explained image refinement and background compositing in post-processing.

The talk then touched on the methods and datasets used in 2D and 3D face generation.

Various metrics have been developed that can be applied to image quality, the synchronization between audio signals and the speaker’s lips, identity preservation and blinking, which Wang outlined.

Challenges remaining in talking face generation range from exercising fine-grained control over facial features like eyes and teeth, head movement and emotion, to generalization of the identity and body. Then there are considerations around forgery detection and social responsibility.

In an example of the first challenge, Wang notes that blinking is related to speech mechanics and thought processes, but the relations are not yet well-understood. Eye-blinking can be generated from target frames or gaussian noise. Some models connect eye movement to overall facial expression, but this method is also still in its early stages of development.

Larger and more diverse datasets could help with generation, according to the SenseTime researcher.

Detection of manipulated video was briefly considered, and deepfake detection was the focus of several other presentations during the event.

Wang sees talking face generation technology improving in the near future, and practical applications expanding as it does.

Article Topics

 |   |   |   |   |   |   |   | 

Latest Biometrics News


Mastercard aims to deliver digital identity, access for 100M in Africa

Mastercard is doubling down on spreading its Community Pass, a digital platform that holds a digital ID and wallet in…


DPI critical to effective digital transformation in government

Robert Opp, chief digital officer of UNDP emphasizes the importance of viewing DPI as the digital equivalent of physical infrastructure,…


Age verification comes to social media as age of unregulated use nears an end

If trends continue, social media is set to follow in the path of cigarettes: an activity benefitting early from lax…


AI can save UK govt £40B annually, revolutionize public services: Blair Institute

AI will soon be used to pre-check applications for correct information, triage cases by calculating complexity and routing them appropriately,…


Papua New Guinea completes national data protection and governance policy

The Ministry of Information and Communications Technology has announced the completion of the national data protection and governance policy, marking…


Interpol issues call for biometrics firm to provide mobile data collection devices

Interpol has issued a tender for mobile ​​biometric collection (MBC) devices for collecting fingerprints and facial images. Specifically, the project…


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Most Read This Week

Featured Company

Biometrics Insight, Opinion

Digital ID In-Depth

Biometrics White Papers

Biometrics Events