What is Open AI's Sora? How it Works, Use Cases, Alternatives & More

What is Sora?

Sora is OpenAI's text-to-video generative AI model. That means you write a text prompt, and it creates a video that matches the description of the prompt. Here's an example from the OpenAI site:

PROMPT: A stylish woman walks down a Tokyo street filled with warm glowing neon and animated city signage. She wears a black leather jacket, a long red dress, and black boots, and carries a black purse. She wears sunglasses and red lipstick. She walks confidently and casually. The street is damp and reflective, creating a mirror effect of the colorful lights. Many pedestrians walk about.

Examples of OpenAI Sora

OpenAI and CEO Sam Altman have been busy sharing examples of Sora in action. We’ve seen a range of different styles, and examples, including:

Sora Animation Examples

PROMPT: A gorgeously rendered papercraft world of a coral reef, rife with colorful fish and sea creatures.

PROMPT: Animated scene features a close-up of a short fluffy monster kneeling beside a melting red candle. The art style is 3D and realistic, with a focus on lighting and texture. The mood of the painting is one of wonder and curiosity, as the monster gazes at the flame with wide eyes and open mouth. Its pose and expression convey a sense of innocence and playfulness, as if it is exploring the world around it for the first time. The use of warm colors and dramatic lighting further enhances the cozy atmosphere of the image.

Sora Cityscape Examples

PROMPT: Beautiful, snowy Tokyo city is bustling. The camera moves through the bustling city street, following several people enjoying the beautiful snowy weather and shopping at nearby stalls. Gorgeous sakura petals are flying through the wind along with snowflakes.

PROMPT: A street-level tour through a futuristic city which in harmony with nature and also simultaneously cyperpunk / high-tech. The city should be clean, with advanced futuristic trams, beautiful fountains, giant holograms everywhere, and robots all over. Have the video be of a human tour guide from the future showing a group of extraterrestial aliens the coolest and most glorious city that humans are capable of building.

Sora Animal Examples

PROMPT: Two golden retrievers podcasting on top of a mountain.

PROMPT: A bicycle race on ocean with different animals as athletes riding the bicycles with drone camera view.

How Does Sora Work?

Like text-to-image generative AI models such as DALL·E 3, StableDiffusion, and Midjourney, Sora is a diffusion model. That means that it starts with each frame of the video consisting of static noise, and uses machine learning to gradually transform the images into something resembling the description in the prompt. Sora videos can be up to 60 seconds long.

Solving temporal consistency

One area of innovation in Sora is that it considers several video frames at once, which solves the problem of keeping objects consistent when they move in and out of view. In the following video, notice that the kangaroo's hand moves out of the shot several times, and when it returns, the hand looks the same as before. 

PROMPT: A cartoon kangaroo disco dances.

Combining diffusion and transformer models

Sora combines the use of a diffusion model with a transformer architecture, as used by GPT.

When combining these two model types, Jack Qiao noted that "diffusion models are great at generating low-level texture but poor at global composition, while transformers have the opposite problem." That is, you want a GPT-like transformer model to determine the high-level layout of the video frames and a diffusion model to create the details.

In a technical article on the implementation of Sora, OpenAI provides a high-level description of how this combination works. In diffusion models, images are broken down into smaller rectangular "patches." For video, these patches are three-dimensional because they persist through time. Patches can be thought of as the equivalent of "tokens" in large language models: rather than being a component of a sentence, they are a component of a set of images. The transformer part of the model organizes the patches, and the diffusion part of the model generates the content for each patch.

Another quirk of this hybrid architecture is that to make video generation computationally feasible, the process of creating patches uses a dimensionality reduction step so that computation does not need to happen on every single pixel for every single frame.

Increasing Fidelity of Video with Recaptioning

To faithfully capture the essence of the user's prompt, Sora uses a recaptioning technique that is also available in DALL·E 3. This means that before any video is created, GPT is used to rewrite the user prompt to include a lot more detail. Essentially, it's a form of automatic prompt engineering.

What are the Limitations of Sora?

OpenAI notes several limitations of the current version of Sora. Sora does not have an implicit understanding of physics, and so "real-world" physical rules may not always be adhered to.

One example of this is that the model does not understand cause and effect. For example, in the following video of an explosion on a basketball hoop, after the hoop explodes, the net appears to be restored.

PROMPT: Basketball through hoop then explodes.

Similarly, the spatial position of objects may shift unnaturally. In the following video of wolf pups, animals appear spontaneously, and the position of the wolves sometimes overlaps.

PROMPT: Five gray wolf pups frolicking and chasing each other around a remote gravel road, surrounded by grass. The pups run and leap, chasing each other, and nipping at each other, playing.

Unanswered questions on reliability

The reliability of Sora is currently unclear. All the examples from OpenAI are very high quality, but it is unclear how much cherry-picking was involved. When using text-to-image tools, it is common to create ten or twenty images then choose the best one. It is unclear how many images the OpenAI team generated in order to get the videos shown in their announcement article. If you need to generate hundreds or thousands of videos to get a single usable video, that would be an impediment to adoption. To answer this question, we must wait until the tool is widely available.

What are the Use Cases of Sora?

Sora can be used to create videos from scratch or extend existing videos to make them longer. It can also fill in missing frames from videos.

In the same way that text-to-image generative AI tools have made it dramatically easier to create images without technical image editing expertise, Sora promises to make it easier to create videos without image editing experience. Here are some key use cases.

Social media

Sora can be used to create short-form videos for social media platforms like TikTok, Instagram Reels, and YouTube Shorts. Content that is difficult or impossible to film is especially suitable. For example, this scene of Lagos in 2056 would be technically difficult to film for a social post but is easy to create using Sora.

PROMPT: A beautiful homemade video showing the people of Lagos, Nigeria in the year 2056. Shot with a mobile phone camera.

Advertising and marketing

Creating adverts, promotional videos, and product demos is traditionally expensive. Text-to-video AI tools like Sora promise to make this process much cheaper. In the following example, a tourist board wanting to promote the Big Sur region of California could rent a drone to take aerial footage of the location, or they could use AI, saving time and money.

PROMPT: Drone view of waves crashing against the rugged cliffs along Big Sur’s garay point beach. The crashing blue waters create white-tipped waves, while the golden light of the setting sun illuminates the rocky shore. A small island with a lighthouse sits in the distance, and green shrubbery covers the cliff’s edge. The steep drop from the road down to the beach is a dramatic feat, with the cliff’s edges jutting out over the sea. This is a view that captures the raw beauty of the coast and the rugged landscape of the Pacific Coast Highway.

Prototyping and concept visualization

Even if AI video isn't used in a final product, it can be helpful for demonstrating ideas quickly. Filmmakers can use AI for mockups of scenes before they shoot them, and designers can create videos of products before they build them. In the following example, a toy company could generate an AI mockup of a new pirate ship toy before committing to creating them at scale.

PROMPT: Photorealistic closeup video of two pirate ships battling each other as they sail inside a cup of coffee.

Synthetic data generation

Synthetic data is often used for cases where privacy or feasibility concerns prevent real data from being used. For numeric data, common use cases are for financial data and personally identifiable information. Access to these datasets must be tightly controlled, but you can create synthetic data with similar properties to make available to the public.

One use of synthetic video data is for training computer vision systems. As I wrote in 2022, the US Air Force uses synthetic data to improve the performance of its computer vision systems for unmanned aerial vehicles to detect buildings and vehicles at nighttime and in bad weather. Tools such as Sora make this process much cheaper and more accessible for a wider audience.

What are the Risks of Sora?

The product is new, so the risks are not fully described yet, but they will likely be similar to those of text-to-image models.

Generation of harmful content

Without guardrails in place, Sora has the power to generate unsavory or inappropriate content, including videos containing violence, gore, sexually explicit material, derogatory depictions of groups of people, and other hate imagery, and promotion or glorification of illegal activities.

What constitutes inappropriate content varies a lot depending on the user (consider a child using Sora versus an adult) and the context of the video generation (a video warning about the dangers of fireworks could easily become gory in an educational way).

Misinformation and disinformation

Based on the example videos shared by OpenAI, one of Sora's strengths is its ability to create fantastical scenes that couldn't exist in real life. This strength also makes it possible to create "deepfake" videos where real people or situations are changed into something that isn't true.

When this content is presented as truth, either accidentally (misinformation) or deliberately (disinformation), it can cause problems.

As Eske Montoya Martinez van Egerschot, Chief AI Governance and Ethics Officer at DigiDiplomacy, wrote, "AI is reshaping campaign strategies, voter engagement, and the very fabric of electoral integrity."

Convincing-but-fake AI videos of politicians or adversaries of politicians have the power to "strategically disseminate false narratives and target legitimate sources with harassment, aiming to undermine confidence in public institutions and foster animosity towards various nations and groups of people".

In a year containing many important elections from Taiwan to India to the United States, this has widespread consequences.

Biases and stereotypes

The output of generative AI models is highly dependent on the data it was trained on. That means that cultural biases or stereotypes in the training data can result in the same issues in the resulting videos. As Joy Buolamwini discussed in the Fighting For Algorithmic Justice episode of DataFramed, biases in images can have severe consequences in hiring and policing.

How Can I Access Sora?

Sora is currently only available to "red team" researchers. That is, experts who are given the task of trying to identify problems with the model. For example, they will try to generate content with some of the risks identified in the previous section so OpenAI can mitigate the problems before releasing Sora to the public.

OpenAI has not yet specified a public release date for Sora, though it is likely to be some time in 2024.

What Are the Alternatives to Sora?

There are several high-profile alternatives to Sora that allow users to create video content from text. These include:

• Runway-Gen-2. The highest-profile alternative to OpenAI Sora is Runway Gen-2. Like Sora, this is a text-to-video generative AI, and it is currently available on web and mobile.
• Lumiere. Google recently announced Lumiere, which is currently available as an extension to the PyTorch deep-learning Python framework.
• Make-a-Video. Meta announced Make-a-Video in 2022; again this is available via a PyTorch extension.

There are also several smaller competitors:

• Pictory simplifies the conversion of text into video content, targeting content marketers and educators with its video generation tools.
• Kapwing offers an online platform for creating videos from text, emphasizing ease of use for social media marketers and casual creators.
• Synthesia focuses on creating AI-powered video presentations from text, offering customizable avatar-led videos for business and educational purposes.
• HeyGen aims to simplify video production for product and content marketing, sales outreach, and education.
• Steve AI provides an AI platform that enables generation of videos and animation from Prompt to Video, Script to Video, and Audio to Video.
• Elai focuses on e-learning and corporate training, offering a solution to effortlessly turn instructional content into informative videos

What Does OpenAI Sora Mean for the Future?

There can be little doubt that Sora is ground-breaking. It’s also clear that the potential for this generative model is vast. What are the implications of Sora on the AI industry and the world? We can, of course, only take educated guesses. However, here are some of the ways that Sora may change things, for better or worse.

Short-term implications of OpenAI Sora

Let’s first take a look at the direct, short-term impacts we might see from Sora in the wake of its (likely phased) launch to the public.

A wave of quick wins

In the section above, we’ve already explored some of Sora's potential use cases. Many of these will likely see quick adoption if and when Sora is released for public use. This might include:

• The proliferation of short-form videos for social media and advertising. Expect creators on X (formerly Twitter), TikTok, LinkedIn, and others to up the quality of their content with Sora productions.
• The adoption of Sora for prototyping. Whether it’s demonstrating new products or showcasing proposed architectural developments, Sora could become commonplace for pitching ideas.
• Improved data storytelling. Text-to-video generative AI could give us more vivid data visualization, better simulations of models, and interactive ways to explore and present data. That said, it will be important to see how Sora performs on these types of prompts.
• Better learning resources. With tools like Sora, learning materials could be greatly enhanced. Complicated concepts can be brought to life, while more visual learners have the chance for better learning aids.

A minefield of risks

Of course, as we highlighted previously, such tech comes with a swathe of potential negatives, and it’s imperative that we navigate them. Here are some of the risks we must be alert to:

• The spread of misinformation and disinformation. Collectively, we’ll have to be more discerning of the content we consume, and we’ll need better tools to spot that which is manufactured or manipulated. This is especially important in an election year.
• Copyright infringement. We’ll need to be mindful of how our images and likenesses are used. Legislation and controls may be required to prevent our personal data from being used in ways we’ve not consented to. This debate will most likely first play out as fans start creating videos based on their favorite film franchises—that said, the personal risks are also massive here.
• Regulatory and ethical challenges. The advances in generative AI are already proving difficult for regulators to keep up with, and Sora could exacerbate this issue. We must navigate the appropriate and fair use of Sora without impacting individual liberties or stifling innovation.
• Dependence on technology. Tools like Sora could be seen as a shortcut for many rather than an assistant. People may see it as a replacement for creativity, which could have implications for many industries and the professionals who work in them.

Generative video becomes the next frontier of competition

We’ve already mentioned a couple of alternatives to Sora, but we can expect this list to grow significantly in 2024 and beyond. As we saw with ChatGPT, there is an ever-growing list of alternatives vying for positions and many projects iterating on the open-source LLMs on the market.

Sora may well be the tool that continues to drive innovation and competition in the field of generative AI. Whether it’s through use-specific, fine-tuned models or proprietary tech that’s in direct competition, many of the big players in the industry will likely want a piece of the text-to-video action.

Long-term implications of OpenAI Sora

As the dust begins to settle after the public launch of OpenAI’s Sora, we’ll start to see what the longer-term future holds. As professionals across a host of industries get their hands on the tool, there’ll inevitably be some game-changing uses for Sora. Let’s speculate on what some of these could be:

High-value use cases can be unlocked

It’s possible that Sora (or similar tools) could become a mainstay in several industries:

• Advanced content creation. We could see Sora as a tool to speed up production across fields such as VR and AR, video games, and even traditional entertainment such as TV and movies. Even if it’s not used directly to create such media, it could help to prototype and storyboard ideas.
• Personalised entertainment. Of course, we could see an instance where Sora creates and curates content tailored specifically to the user. Interactive and responsive media that are tailored to an individual’s tastes and preferences could emerge.
• Personalised education. Again, this highly individualized content could find a home in the education sector, helping students learn in a way that’s best suited to their needs.
• Real-time video editing. Video content could be edited or re-produced in real-time to suit different audiences, adapting aspects such as tone, complexity, or even narrative based on viewer preferences or feedback.

The lines between the physical and digital worlds begin to blur

We’ve already touched on virtual reality (VR) and augmented reality (AR), but Sora has the potential to revolutionize how we interact with digital content when combined with these mediums. If future iterations of Sora are able to generate high-quality virtual worlds that can be inhabited within seconds—and leverage generative text & audio to populate it with seemingly real virtual characters—this raises serious questions about what it means to navigate the digital world in the future.

Closing Notes

In conclusion, OpenAI's Sora model promises a leap forward in the quality of generative video. The forthcoming public release and its potential applications across various sectors are highly anticipated. If you’re eager to get started in the world of generative AI, our AI Fundamentals skill track will help you get up to speed with ​​machine learning, deep learning, NLP, generative models, and more.

For more resources on the latest in the world of AI, check out the list below:

• Subscribe to the DataFramed Podcast
• Become an AI Developer Code-Along Series
• Browse our AI tutorials