ByteDance releases new generative AI model OmniHuman

Chinese tech company ByteDance has come up with a generative AI framework that can create highly realistic videos of a human based on a single image and motion signal called OmniHuman-1.

ByteDance’s researchers demonstrated the technology by generating several realistic human videos, including Albert Einstein and Nvidia CEO Jensen Huang. The videos show humans talking and singing in challenging body positions, including using their hands, and in different aspect ratios such as portraits, half-body and full-body. The system can also animate cartoons.

The company behind TikTok says that the framework beats existing technology which is still struggling to scale beyond animating faces or upper bodies, limiting their potential in real applications. OmniHuman outperforms existing methods because it can generate extremely realistic human videos based on weak signal inputs, especially audio, according to a research paper published by the company.

“In OmniHuman, we introduce a multimodality motion conditioning mixed training strategy, allowing the model to benefit from data scaling up of mixed conditioning,” the researchers write. “This overcomes the issue that previous end-to-end approaches faced due to the scarcity of high-quality data.”

The researchers relied on more than 18,000 hours of human-related data for training the framework, allowing it to learn from text, audio, and body movements. This resulted in more natural-looking human videos.

“Our key insight is that incorporating multiple conditioning signals, such as text, audio and pose, during training can significantly reduce data wastage,” says the paper.

The system initially handles each input type independently, condensing movement details from text descriptions, reference images, audio signals and movement data into a compact format. It then progressively enhances this data into realistic video output, refining motion generation by comparing its results with real videos.

ByteDance has been investing in AI video generation, rivaling firms such as Meta, Microsoft and Google DeepMind. In January, the company released an upgrade to its AI model Doubao, claiming it outperforms OpenAI’s o1 benchmark test AIME.

Article Topics

ByteDance  |  deepfakes  |  generative AI  |  OmniHuman-1  |  synthetic data  |  synthetic voice