What Are AI World Models? How They Work and 2026 Trends

Let me start off this article with an analogy. Imagine putting a glass on the edge of a table. If someone bumped the table, the glass might fall and break. But the important part is that you did not need to test it first. 

From what you see and what you already know, you could picture the result and decide to move the glass somewhere safer.

We make these small predictions all the time, but AI systems respond to the information in front of them without having a deeper sense of what might happen next. 

A world model is an attempt to change that. 

It is an approach that gives an AI an internal picture of its surroundings so it can try out possible actions before taking them. 

In very simple terms, it helps the system move beyond “What can I see?” and instead ask, “What will happen if I do this action?”

This way of thinking can be useful for robots, self-driving cars, scientific research, and AI agents. 

So in this article, we will look at what world models are, how they work, and why they are becoming an important area of AI research.

What Are World Models?

Let’s start by defining what a world model is:

A world model is an internal representation of an environment that captures how its states change over time and how they respond to actions, enabling an AI system to predict possible outcomes and plan before acting.

The main idea is that the AI does not just react to each new piece of information on its own. It uses what it has already seen to build up an understanding of the situation. This can help it notice how one outcome can lead to another and, therefore, make better decisions about what to do next.

Let me give you an example. 

Imagine a robot moving around a kitchen. 

It might learn that an open cupboard is an obstacle, a wet floor is slippery, and a person standing in the room can change the route it takes. But crucially, it does not need to remember every single detail of the kitchen, but rather only the details that could affect its next move (state).

Now, of course, the model will not always predict the future correctly, but it can still consider a few possible outcomes before it acts. 

This ability gives the system a chance to avoid a problem instead of only reacting after it happens. 

Before we understand how this works, I want to take a moment and discuss how World Models are different from Large Language Models.

World models vs large language models

Large language models and world models can both learn from huge amounts of data, but they focus on different problems.

A large language model is like a skilled autocomplete system that uses patterns from its training data to predict what should come next. 

That is why it can answer questions, write code, and summarise an article.

A world model adds an understanding of the situation itself. 

Instead of only predicting which words or actions might come next, it helps connect what is happening now in the present with what could happen afterward, allowing it to choose an action based on the situation and its possible results.

I like to think of this comparison when comparing the two - a city guide and a person who knows the city well. 

The guide can tell you about the roads, but the experienced local has a better sense of what will happen if a road is blocked or traffic suddenly builds up. 

A language model is generally better at talking about a situation, but a world model is designed to reason about how the situation may unfold.

An important point to note is that the two can also work together.

A language model might understand, “Take the blue cup to the table,” whereas a world model can help decide whether the cup is reachable, whether the path is clear, and what could go wrong.

In short, large language models focus on what to say next (i.e., what token should be outputted next), whereas world models focus on what might happen next (predicting the next state).

Figure 1: An LLM maps text to the next token; a world model maps state and action to the next state. Original image by the author.

How Do World Models Work?

Now let's get into the actual mechanics of a World Model. 

In this article, I will refer to the original 2018 World Model paper by David Ha and Jürgen Schmidhuber.

They showed how a driving agent could learn a simpler version of the world around it. 

Instead of making decisions directly from every detail in an image, the agent first turned each view of the track into a compact description. 

It then learned how those descriptions changed as the car moved and used that knowledge to choose its next action. 

Because the model could simulate what might happen next, the controller could also be trained in these imagined driving situations rather than only through repeated interaction with the actual track (i.e., the environment).

I believe a good way to understand this is by imagining a driver.

• First, the driver looks at the road and picks out the details that matter, such as the direction of the track and the car’s position.
• Next, the driver will remember what has happened over the last few moments and use that context to anticipate the next turn.
• Finally, based on the two points above, the driver steers.

This is very intuitive, but importantly, the paper follows the same basic pattern:

• It sees
• It remembers
• It acts

A robot moving through a room can use a similar approach, remembering how each movement changes its position before deciding where to go next.

These three jobs are handled by connected parts of the model.

Figure 2: How vision, memory, and controller cycle inside the environment. Original image by author.

It starts with the vision component (V), which turns each camera image into the compact representation used by the rest of the system.

Vision Component (V)

The vision component is the part of the world model that turns a raw image into something the rest of the system can work with. 

In the paper's experiments, the agent received images from the environment, such as frames from the CarRacing task. 

But surely feeding every pixel directly into the decision-making system would make the problem unnecessarily large?

Figure 3: An example frame from the CarRacing environment used in the original World Models experiments. This is the type of raw image that the vision component needs to compress into a much smaller latent representation. Source: Ha and Schmidhuber, World Models (2018).

The solution is using a variational autoencoder, or VAE, to compress each image into a short latent vector called z.

The VAE, a type of Autoencoder, works a lot like the driver glancing at the road. 

A driver does not need to memorize the color of every blade of grass or every mark on the edge of the track, but rather important details such as where the road bends, where the car is positioned, and what parts of the scene may affect the next move (such as a pothole). 

In the same way, the VAE learns to keep a useful summary of the frame while leaving out visual details that are less important for driving.

I would like to mention that the compressed vector (z) is not a written description like “the car is approaching a left turn" but rather, is a numerical representation learned from the images themselves. 

The VAE is trained to reconstruct the original frame from this representation, encouraging z to retain enough information about the scene to recreate it. 

Once our VAE is properly trained, it can reduce a new camera image to a compact form before passing it to the next part of the model.

Figure 4: Encoder, latent vector z and decoder reconstructing a CarRacing frame. Original image by author.

For our car-driving agent, this means that the memory component does not have to process a full stream of complicated images. 

It can instead follow a sequence of smaller representations - one describing the track and the car now, another describing them a time step later, and so on.

This sequence is what allows the model to learn how the scene changes over time, leading naturally to the role of the memory component.

Memory Component (M)

By this point, you should know what the memory component roughly does - it gives our model a sense of time. 

A single image can show where the car is, but cannot explain whether the car is turning, drifting, or about to leave the track. 

Another example is that an image can give you the position, but to capture velocity, we need more than 1 image. 

Therefore, the model needs to compare the current view with what it saw moments earlier.

In the original paper, Ha and Schmidhuber used a model called an MDN-RNN for this job. 

An MDN-RNN is simply the particular memory mechanism they chose. Later world models now use GRUs, LSTMs, Transformers, Mamba, or something else (remember Transformers were not that popular in 2018!)

What matters here is that the model keeps track of what happened earlier and uses that information to understand what may happen next. 

At each step, it received:

• VAE’s latent vector z
• Action the car had just taken a_t
• Previous hidden state h_{t-1}.

Using these, it then updates that running memory and predicts the next latent state.

Figure 5: The structure of an MDN-RNN unrolled over time. At each step, the RNN uses the current latent state, previous hidden state and action to update its memory, while the MDN predicts the next possible latent state. Source: ResearchGate – Structure of an unrolled MDN-RNN.

The mixture-density part allows it to represent several possible futures rather than committing to one exact prediction (no need to understand the core mechanics, as this part, like most other parts, is replaceable). 

For example, on the track, a small steering change could lead to slightly different positions on the next frame, depending on the car’s speed and where it already was.

Therefore, the important point is the memory itself:

The memory carries forward information that may no longer be visible in the current frame.

This memory is also what makes the model useful for imagining possible outcomes. 

Rather than waiting for the real car to take every action (which is extremely inefficient), the system can use its learned dynamics to estimate what might happen after steering left, steering right, or continuing forward. 

Those predictions are passed to the controller, which turns the model’s understanding of the situation into an action.

Controller Component (C)

Now we reach the third component - the controller, which is the decision-making part of the system. 

It receives the current latent representation z from the vision component and the hidden state h from the memory component. 

Together, these give it both a description of what the car can currently see and a record of how the situation has been unfolding. 

The controller then produces the next action, such as steering, accelerating, or braking.

In the CarRacing task, this was intentionally a small and simple policy rather than another large network. 

It did not need to understand raw pixels or remember the whole drive by itself. 

Why? 

Because the VAE had already reduced the image to z, and the MDN-RNN had already gathered the recent sequence into h. 

The controller’s only job was to use those two pieces of information to decide what the car should do next, very similar to a driver looking at the road while also remembering that the car has just come out of a bend.

Now, one of the paper’s important ideas was that the controller could be trained inside the learned world model. 

Instead of testing every decision on the real track, the system could let the MDN-RNN predict how the car’s situation would change, then see whether a proposed sequence of actions led to good results. 

Thus, the controller sits at the end of the vision–memory–action process, and it is this separation that sets up the next step - training the world model and its controller from experience. Let's look into this in a bit more detail.

Training a World Model

The first thing I would like to mention is that the system was not taught to see, remember, and control the car all at once. 

Instead, the researchers trained each part separately and then connected them together. This made the process easier to manage because each model had a clear job to learn from the driving data.

Phase 1: Teaching the VAE to see

The process began with the VAE. The researchers collected images from the CarRacing environment and used them to teach the VAE how to compress each frame. 

The encoder took an image of the track and turned it into the smaller latent vector z. The decoder then tried to rebuild the original image from z. 

If the reconstructed image was different from the original, the VAE adjusted itself and tried again. 

After seeing many frames, it learned to keep the details that mattered for driving, such as the position of the car and the shape of the track, without having to pass every pixel to the next part of the system.

Phase 2: Teaching the memory component to predict

Once the VAE could create these latent descriptions, the researchers trained the MDN-RNN. 

This time, the training examples were sequences from the car’s experience. 

At each step, the model received the current latent vector, the action that was taken, and the reward that followed. 

It then tried to predict the next latent vector and the next reward. The RNN carried information forward from earlier steps, while the mixture-density network represented several possible next states instead of assuming that the future would always be exactly the same. 

This was important because the same action could lead to slightly different outcomes depending on what had happened before.

Phase 3: Training the controller in "dream" mode

With the visual and memory parts trained, the researchers moved on to the controller. 

The controller was deliberately kept small. 

It received z from the VAE and h from the MDN-RNN, and its parameters determined whether the car should steer, accelerate, or brake. 

Rather than improving it through ordinary backpropagation through the entire game, the researchers tested different controller parameters and kept the versions that produced better driving behavior. 

The controller could be tested in the model’s imagined environment, where the MDN-RNN predicted what would happen after each action.

Now here is the crucial step: this allowed many candidate controllers to be compared without carrying out every trial on the real track.

This final stage is where the separate pieces became one agent:

• The VAE supplied a compact view of the current frame
• The MDN-RNN used that view to maintain a short-term memory and predict what might happen next
• The controller used both to choose an action.

Figure 6: The full World Models architecture, showing how the VAE (V), MDN-RNN (M) and controller (C) work together with the environment. The VAE compresses observations into a latent state, the MDN-RNN maintains the model's memory, and the controller uses both to choose the next action. Source: Ha and Schmidhuber, World Models (2018).

In this way, the world model was trained from recorded experience first, while the controller learned to make use of it afterward. 

The agent could then practice possible decisions in its learned version of the environment before acting in the real one.

Key World Models in 2026

Now, let's talk about how the industry is applying World Models in different scenarios.

Google DeepMind's Genie 3

Let me start off by explaining what makes Genie 3 different. 

DeepMind introduced it in August 2025, and instead of just generating a video that you watch, it creates a world you can actually move through in real time. 

You type a prompt, and as you (or an AI agent) explore, the model predicts what should appear next and tries to keep everything consistent. 

It can run at 20-24 frames per second at 720p, but, and this is an important limitation, only for a few minutes at a time. 

So why does this matter?

Well, it means an agent could practice navigating somewhere unfamiliar or reacting to a sudden change before ever attempting it in the real world. 

This also allows researchers to create situations that would otherwise be too expensive, dangerous, or rare to collect. 

But I should mention that it is still very much a research preview - the environments can lose consistency or drift away from real-world physics, which is a real limitation for precise robotics work. 

Even so, Genie 3 is a great example of AI moving past generating isolated images and videos and towards building interactive worlds in which an agent can practice decisions.

Figure 7: An example of an interactive world generated by Genie 3. Rather than generating a fixed video, Genie 3 generates the world as the user moves through it. Source: Google DeepMind, Genie 3: A New Frontier for World Models.

NVIDIA Cosmos

NVIDIA took a slightly different approach with Cosmos, which was announced at CES in January 2025. 

Rather than a single model, Cosmos is best thought of as a whole platform for physical AI:

• A combination of generative models
• Video tokenizers
• Data-processing tools and safety features, aimed mainly at robotics and self-driving.

Figure 8: An overview of NVIDIA Cosmos and the different components that make up its physical AI platform. Source: NVIDIA, Advancing Physical AI with NVIDIA Cosmos.

The core idea is that the models learn from real and simulated video to predict plausible future scenes, often guided by a specific action, so that a team can see how an environment might respond before deploying anything in the real world. 

The initial family had three parts:

NVIDIA released the model weights openly, but they are upfront that this is not a replacement for real-world testing, as generated scenes can still be visually or physically inaccurate, and staying consistent over long sequences remains difficult.

World Labs (Fei-Fei Li)

World Labs was founded by Fei-Fei Li (a very well-known computer-vision researcher) along with several others, and came out of stealth in September 2024, having raised $230 million. 

Their whole focus is on something they call "spatial intelligence," which is really just their term for AI systems that understand the structure and geometry of a place, rather than treating it as a flat collection of images or words. 

Their first product, Marble, is a multimodal world model that can generate 3D environments from text, images, video or even a rough 3D layout. Importantly, these environments are designed to be spatially consistent and navigable, so you can actually explore them, which is why they are useful for things like visualisation, game development, virtual production and simulation. More broadly,

Figure 9: An example of a 3D world generated using Marble. The generated environment is designed to be spatially consistent so that a user can actually move around and explore it. Taken from: World Labs, Marble: A Multimodal World Model.

AMI Labs (Yann LeCun)

AMI Labs, short for Advanced Machine Intelligence Labs, is a Paris-based startup founded by Yann LeCun after he left his role as chief AI scientist at Meta. 

He has argued for a while now that large language models alone are unlikely to get us to human-level intelligence, simply because they never build a rich enough understanding of the physical world. 

AMI Labs is his attempt at an alternative - systems that can learn internal representations of reality and use predictions about how situations might unfold to plan and act. 

What really stood out to me was the funding - $1.03 billion in seed funding announced in March 2026, at a pre-money valuation of $3.5 billion. 

But I want to be clear, it's still a young, research-focused company with nothing shipped yet, so its significance right now lies more in how much people are willing to bet on the idea than in any proven results.

Applications of World Models

Hopefully, you will agree with me that World Models are being used in so many different and exciting applications. 

I have created a larger list of applications here:

World Models Challenges and Limitations

A really big limitation is that world models are still difficult to build and train. 

They need large amounts of useful data, and that data does not always show every situation a system might face. A small mistake in the model’s prediction can also grow over time, especially when it is planning several steps ahead. 

This matters in areas such as robotics and self-driving cars, where an incorrect prediction could lead to real damage.

There is also a limit to how much a world model can understand. It may learn patterns from its training data without understanding the full meaning behind them, and unusual situations can still confuse it. 

World models can be useful for testing ideas and predicting possible outcomes, but they should not be treated as perfect copies of the real world. 

However, human supervision and real-world testing are still important. 

Final Thoughts

In conclusion, world models give AI a way to learn how environments change and to think ahead before taking an action. 

Again, I would like to reiterate that they are not perfect copies of the real world, but they can help systems practice, make better decisions, and handle complex tasks more safely. 

As these models improve, they will become an important part of robotics, autonomous vehicles, and other AI systems.