ByteDance's OmniHuman: A Guide With Examples

What Is Omnihuman?

OmniHuman is an image-to-video generation model that can generate realistic videos or animations based on an image. Technically, its full name is OmniHuman-1, suggesting it’s part of a longer-term project with future versions in development. For convenience, I’ll refer to it as OmniHuman throughout this blog.

Judging by the examples offered by the research team behind OmniHuman, the model excels at animating the subject in a way that appears to move naturally, perform gestures, and even sing or play instruments.

OmniHuman can generate videos with different input sizes and body proportions, supporting various types of shots, such as close-ups, half-body, or full-body. It can also perform lip sync with audio.

Note that the input images for most video examples in this article are the first frame of each video (plus the audio). This is important to keep in mind to get a better idea of how easy it is to generate these videos using OmniHuman.

OmniHuman Features

Support for a wide range of subjects

OmniHuman can handle a diverse range of inputs beyond just human figures. This includes cartoons, artificial objects, animals, and even those tricky poses that may challenge traditional video creation tools.

OmniHuman also supports multiple aspect ratios, which is sometimes a limitation of video generation models. The video above has a portrait (9:16) aspect ratio, while the one below has a square (1:1) aspect ratio.

Talking and singing

In the example below, we see a realistic AI-generated Ted Talk. To me, it’s wild to think that this was generated from a single image. The body movements are quite convincing and consistent with the speech.

In this second example, we have an example of a singing subject. This example is less convincing because the guitar hand movement doesn’t match the guitar song.

Lip sync

The next example really shows how strong OmniHuman is when it comes to lip sync. Unlike the guitar hand motion, this video delivers a truly believable performance as the person really appears to be singing, even being consistent with the pitch.

This is also true with regular speech, not just singing (see example below). The main downside of the video below is that I can see some artifacts around the hair when the kid moves. Also, the color of the lips and whiteness of the teeth are very unnatural and don’t match the subject.

Full body, half body, and closeups

These next two examples showcase OmniHuman’s ability to generate half-body videos as well as closeup ones. Let’s start with the half-body example:

And now, let’s see a video generated for a close-up:

Animating hands

One of the things that video and image generation models often struggle with is hands. For some reason, hands often pose a big challenge for AI, resulting in extra fingers and glitches. From their examples, OmniHuman seems to deal with these quite well.

It seems to also be able to handle cases where an object is being held:

Video driving

We’ve seen that OmniHuman supports audio driving where audio is used to guide the video generation to match with it. However, OmniHuman also supports video input for video driving. In this way, it can mimic specific video actions.

The reason why OmniHuman can support both audio driving (making the video consistent with a given audio) and video driving is the way it was trained, which we will explore next.

How to Access OmniHuman?

At the time of publishing this article, detailed information about accessing Omnihuman isn't available. For official updates or announcements on the release and access details, keep an eye on ByteDance's official channels, such as press releases or their corporate website. Additionally, since ByteDance owns TikTok, updates might also appear on platforms associated with the company.

How Does OmniHuman Work?

OmniHuman gets its name from the fact that, contrary to current models, it integrates multiple condition signals during the training phase, which they call omni-conditions training. In simple terms, these condition signals refer to different types of information that are used to guide the creation of a video of a human.

Current models often rely on single conditioning signals, like audio or pose. For example, audio-conditioned models focus on facial expressions and lip synchronization, while pose-conditioned models emphasize full-body poses.

However, not all data is perfectly aligned with these specific signals. As a result, large amounts of potentially useful data are discarded during filtering processes because they contain elements (e.g., body movements unrelated to speech in audio-driven models) that don't fit the narrow scope of the conditioning signal.

Imagine you're trying to create an animation of a person, like in a video game or a cartoon. To make the animation look realistic, you need to know more than just how the person looks in a single picture. You also need details about how they move, what they're saying, and even the poses they might strike.

OmniHuman combines three types of conditions to learn to generate videos:

• Text: This means using written words or descriptions to help guide the animation. For example, if the text says, "The person is waving their hand," the animation uses this information to make the person wave.
• Audio: This is sound, like someone's voice or background music. If the person in the animation is saying something, the model uses the audio to make sure their lips move correctly to match the words.
• Pose: This refers to the position and movement of the person's body. For instance, if you want to animate someone dancing, the poses provide a guide to how their arms and legs should move.

The idea here is that by combining these different signals, the model can create videos that look very realistic.

Another advantage of omni-conditions training is that the model can reduce data wastage compared to other models. Other models aren’t able to fully take advantage of the data that’s used to train them for the following reasons:

• Specificity of conditioning signals: Current models often rely on single conditioning signals, like audio or pose. For example, audio-conditioned models focus on facial expressions and lip synchronization, while pose-conditioned models emphasize full-body poses. However, not all data is perfectly aligned with these specific signals. As a result, large amounts of potentially useful data are discarded during filtering processes because they contain elements (e.g., body movements unrelated to speech in audio-driven models) that don't fit the narrow scope of the conditioning signal.
• Data filtering and cleaning: To improve training efficiency and model accuracy, existing methods apply rigorous data filtering and cleaning processes. For example, audio-conditioned models filter data based on lip-sync accuracy, while pose-conditioned models filter for pose visibility and stability. These processes remove data that might contain useful motion patterns and diverse scenarios needed for expanding the model's capabilities.
• Limited applicability: Due to their reliance on highly curated datasets, these models are only applicable to a narrow range of scenarios—such as front-facing, static backgrounds. This restricts the generalization capabilities of the models in more diverse, real-world scenarios.

By using omni-conditions training, the OmniHuman model can effectively use larger and more diverse datasets, resulting in more realistic and flexible human video generation across a wide range of conditions and styles.

Training Data for OmniHuman

The dataset curated for training OmniHuman comprises approximately 18.7K hours of human-related data, selected using criteria essential for video generation, such as aesthetics, image quality, and motion amplitude.

Of this enormous dataset, 13% was earmarked for training with audio and pose modalities based on stringent lip sync accuracy and pose visibility conditions. This dual-layered approach ensures that only the most relevant data informs the model's understanding of human animation, allowing it to perform effectively across varied scenarios.

Traditional models have often trained on much smaller datasets, typically involving hundreds of hours or even less, focusing narrowly on specific body parts or types of animation (e.g., facial animations or full-body poses) under rigid scene constraints. This limited the generalizability and applicability of these models across different tasks. By avoiding excessive filtering and embracing weaker conditioning tasks along with their respective data, OmniHuman mitigates the limitations imposed by exclusive reliance on highly filtered datasets.

Moreover, unlike typical end-to-end, single-condition models, OmniHuman employs its omni-conditions training strategy to utilize mixed data training, thus overcoming inherent challenges seen in other leading works that focused on using very specialized videos to train the model to generate specific types of videos. These models do not exhibit the versatility seen in OmniHuman.

Use Cases of OmniHuman

Let’s explore a few of the use cases OmniHuman could have. As with everything, there’s always a good and a bad side.

Positive use cases

Here are a few examples of positive use cases for OmniHuman:

• Content creation and engagement: This kind of technology has tremendous value for TikTok and other social media platforms. I can already see OmniHuman implemented as a feature in TikTok.
• Marketing and advertising: Crafting personalized and immersive ads with lifelike characters.
• Democratization of film creation: AI makes video creation much easier. This will enable creative individuals who lack the technical skills, budget, or equipment to bring their ideas to life.
• Entertainment and media: Hollywood could use this kind of technology to revive deceased actors for new roles in films.
• Bringing historical figures back to life: One of their examples shows a video of Einstein making a speech about art. Even though I knew it wasn't real, I felt something for seeing Einstein come to life. I could see this being very engaging if used in a lecture about the Theory of Relativity. We could also imagine a museum adding this kind of experience with other historical figures.

Negative use cases

Despite the positives, OmniHuman can also be a dangerous tool and can lead to many problems:

• Misinformation and political manipulation: Fabricating videos of political leaders to stir governmental disruption or electoral chaos.
• Financial fraud: Creating fake endorsements by celebrities to promote scams or fraudulent investments. There’s recently been a case of a French woman who lost about $850,000 because of a deepfake celebrity scam.
• Invasion of privacy: Unauthorized use of personal images to create videos without consent.
• Identity theft and social engineering: Impersonating individuals to conduct malicious activities or scams.
• Reputation damage and defamation: Producing fake videos designed to harm individuals' reputations or careers.
• Unethical content use: Using the technology to place individuals' likenesses in adult content or other objectionable material without consent.
• Corporate espionage and market manipulation: Creating videos of business leaders for unethical practices like insider trading.

Risks and Ethical Concerns of OmniHuman

We suggested some of the negative use cases that OmniHuman could have. I believe that the biggest concern with OmniHuman is its potential to trivialize the production of deepfake videos that appear real but are completely fabricated. 

As we mentioned, this poses a threat, for example, in politics, where fake videos can be used to spread false information about politicians or influence public opinion during elections. For example, a deepfake might show a politician saying something they never said, leading to confusion and mistrust among voters.

However, this isn’t a problem specific to OmniHuman, as these are already happening. But I worry about how much worse it would get if anyone could create a deepfake with a click of a button using it.

A survey by Jumeo, an ID verification firm, found that 60% of people encountered a deepfake in the past year, indicating that such content is becoming more widespread.

The same survey revealed that 72% of respondents were worried about being fooled by deepfakes on a daily basis. This suggests a significant level of concern among the public about being deceived by AI-generated content.

This report from Deloitte shows that AI-generated content was linked to more than $12 billion in fraud losses in 2023, with projections suggesting it could reach $40 billion in the U.S. by 2027. This underscores the financial risks associated with deepfake technology being used in scams.

These risks demand robust regulatory frameworks and effective detection tools to mitigate potential misuse. As OmniHuman and similar technologies evolve, it becomes increasingly critical to balance innovation with responsibility, ensuring that such powerful tools are wielded conscientiously.

Conclusion

Assuming the examples provided by the OmniHuman research team were not cherry-picked, this video generation tool has the potential to transform digital content creation across various industries. By integrating multiple conditioning signals—such as text, audio, and pose—OmniHuman generates highly realistic and dynamic videos, setting a new standard in authenticity and versatility.

However, while OmniHuman’s capabilities are impressive, they also raise significant ethical and societal concerns. The ease with which this technology can create realistic deep fakes adds fuel to already existing issues around misinformation, fraud, and privacy invasion.