Research on AI as a director still very far away from generating an Oscar

New research shows that digitally changing the expression of someone captured on video is getting better, although that is a relative accomplishment.

English and Greek scientists have published a paper, which has not been peer-reviewed, announcing their so-called neural emotion director, which works on images recorded in the wild.

While related to controversial work on emotion recognition, this development seemingly enables someone to impose one of six basic emotions (plus neutral) on a video subject, one at a time. The AI can use an external reference style or just a semantic emotional label.

It does not require example images of the subject from a training dataset, according to the researchers, who are from the University of Exeter, the National Technical University of Athens and Greece’s Foundation for Research & Technology – Hellas.

They used a parametric 3D face representation of a subject that can be disentangled from the person’s head and actual expressions.

They created a deep domain translation framework capable of altering expressions in a believable way.

Then, a neural face renderer is used to manipulate the face in the original video file.

The paper offers numerous examples employing video frames of, primarily, highly accomplished American actors. In retrospect, that might not have been the best decision as each is famous for his or her ability to visibly emote, often with subtlety.

At worst, frames of the result are grotesque. The actors often look as if they are trying to talk around a stranger’s dentures in their mouths. Mostly, the effect is ambivalent.

In the case of actual gif-length video clips, it is always obvious that something is not right.

The report’s authors compare their work to past work of other scientists, to show the superiority of their method, and they do not exaggerate.

The sight of actress Julia Roberts, processed via GANmut, seemingly with watery black ink running from her mouth and eyes is a disconcerting clue. Director Quentin Tarantino’s bulging left eye appears to be trying to escape his face in another GANmut example.

Article Topics

AI  |  biometrics  |  deepfakes  |  emotion recognition  |  research and development