Vision-Language-Action Models Explained: How Robots Learn to See, Understand, and Act

Imagine you tell someone, “Grab the mug and put it on the shelf.” They do not need a diagram; they just look at the mug, understand the sentence, and their hand goes there.

Three different things occurred nearly at the same time here. Seeing, understanding, and moving happen almost together for that person, not as three separate steps handed between different parts of their brain. 

For most of robotic AI’s history, that was not how machines worked at all.

In the past, we have had vision-language models that can look at a messy workbench and describe exactly what was on it, including finer details such as the brand name of a tool.

Classical robot controllers could pick up that screwdriver reliably, as long as nobody moved it two centimeters to the left. The two halves of the problem, seeing and understanding on one side and acting on the other, developed almost entirely separately.

This is where Vision-language-action (VLA) models come in – they are an attempt to close that gap.

In this article, we will look at what VLA models are and how their three-part architecture works. We’ll explore how they differ from vision-language models (VLMs) and from classical robot control, and over the models that actually matter in 2026, and where they are used. Finally, we’ll look at what CLA modes still get wrong. 

What Are Vision-Language-Action (VLA) Models?

Let’s start off by formally defining what VLA models are:

A vision-language-action model is an AI system that processes a camera feed, a natural language instruction, and the robot’s own state, then outputs motor commands, with its vision, language, and action modules trained jointly and run in one forward pass rather than as a classical perception-then-planning-then-control pipeline.

The main thing is that perception, language, and action are not separate systems stitched together with hand-written glue code. 

They are submodules trained jointly. So, a vision or language error that would have derailed a classical pipeline’s handoff gets corrected by gradient descent rather than caught, or missed, by a hard-coded interface.

Three boxes and two brittle handoffs on top, one fused model on the bottom. This is the actual structural difference I keep talking about. Image by Author.

I like to put the distinction this way: a VLM is a brain with no body, and a VLA gives it one. 

The perception is not a separate service the robot calls out to, but rather, it is wired into the jointly trained pipeline that decides how far to move the gripper.

So far, I have mentioned VLM as well, so let's compare these 2 architectures.

VLA vs. Vision-Language Model (VLM)

A VLM (Vision-Language Model) takes in images and text and produces text or embeddings.

 A VLA (Vision-Language-Action Model) on the other hand, takes the same kind of input and produces an action instead. 

That sounds like a small change until you see what it does downstream:

You might have heard of models such as GPT-4V, Gemini, or LLaVA before. 

A very common assumption is that a VLA is just one of those with a robot arm attached.

However, it is not that simple. 

The model has to learn an entirely new output space, one where a token corresponds to a real physical quantity, such as end-effector displacement or gripper aperture, and where getting it wrong knocks a cup off a table rather than producing a slightly odd sentence.

If you want to see how GPT-4V, Gemini, and LLaVA stack up against the rest of the field, DataCamp’s Top 10 Vision Language Models in 2026 rounds up the current leaders.

There is a data-side consequence, too. 

A VLM can train on billions of image-text pairs because the internet supplies them for free. 

Robot trajectories, however, have to be collected by an actual robot performing an actual task, usually under teleoperation, which is exactly why the largest open robotics dataset, Open X-Embodiment, tops out at around 1 million trajectories rather than 1 billion.

Manually collecting such trajectories is also very expensive and slow.

22 embodiments, one dataset, and still under 1 million trajectories total. This is what I mean when I say robot data does not scale the way web text does. Source: Open X-Embodiment project page.

VLA vs classical robot control

Now, before talking about how VLA Architecture works, I want to also explore and compare it with classical robot control.

Classical robot control is built for one task in one place, under conditions that somebody locked down in advance. 

Take a vending machine arm, for example. 

It grabs the same snack from the same shelf in the same spot every time because the lighting, object positions, and required motions do not change between picks.

That narrow, unchanging setup is exactly what lets the control system be rigid and reliable.

But now let's change the internal configuration of the vending machine, or change the lighting, or drop an item into the scene that was never part of the plan. 

Now the controller will completely fail as it has never seen the environment like this.

A VLA generalizes better to conditions it was never explicitly trained on, and the reason is that the underlying VLM backbone already knows what a mug, a towel, or a drawer looks like from web-scale pretraining before it ever saw a robot demonstration.

Fine-tuning it on a hundred examples of folding towels allows it to learn how to move given one it already recognizes. 

That transfer is the actual breakthrough here, not raw precision, and I would like to mention that a well-tuned classical controller still can beat a VLA on precision in a fixed, controlled setting.

How Do VLA Models Work?

Let's dive deep into the architecture. 

For this article, I have chosen to go with the OpenVLA Architecture as it is easy to understand.

This is the pipeline I describe below: DINOv2 and SigLIP feeding a Llama 2 backbone, with the action decoder predicting discretized tokens on the other end. Source: OpenVLA project page.

A VLA has three separate modules: 

• A vision encoder
• A language backbone 
• An action decoder

These are trained end-to-end together. 

The vision and language representations get fused, and this combined representation is what the action decoder turns into motor commands. 

Importantly, the three parts are tightly coupled rather than pipelined, which is the whole design decision that separates a VLA from a VLM bolted onto a controller.

If it helps, think of it the same way I described the vision, memory, and controller split in the original World Models paper: it sees, it understands, it acts. 

The difference here is that a VLA learns all three jobs together, end to end, from demonstrations, rather than training each piece separately and stitching them together afterward.

Let’s now explore each of them individually.

Vision encoder: what the model sees

The vision encoder turns a raw camera frame into a dense representation that the rest of the network can actually use. 

Modern VLAs use pretrained vision transformers such as DINOv2 or SigLIP, and quite often both at once.

OpenVLA is a good example of using both. 

It fuses DINOv2, which is strong on spatial and geometric detail, with SigLIP, which is stronger on semantic features that line up well with language. 

The two sets of features get concatenated per image patch and projected into the language model’s embedding space (in this case, it is LLama 2 7B).

I would like to mention that using more than one camera is standard in real deployments. 

For example, a wrist-mounted camera can give close-range detail during the final approach to an object, an overhead camera can give the scene layout, and a side view helps with depth that a single camera cannot recover on its own. 

pi0 (more about this below) takes multiple camera views plus the robot’s proprioceptive state.

Please note that state input matters more than it sounds, as a picture of an arm does not tell you what torque is currently in the joints.

Language encoder: what the model is told

The language side encodes an instruction, something like “pick up the red block and place it in the blue bin,” into a representation that lines up with the visual features in the same space. 

Nearly every VLA reuses a pretrained large language model (LLM) backbone rather than training a text encoder from nothing.

In our case, OpenVLA used Llama 2 7B. pi0 and pi0.5 built on PaliGemma, a 3-billion-parameter VLM. SmolVLA used a modified SmolVLM2 stack.

Reusing an LLM this way is really about inheritance. 

You get instruction following and compositional understanding, which is why a VLA can often handle “put the fruit in the bowl” without ever having seen that exact phrase during robot training. 

Action decoder: what the model does

The action decoder is where the fused representation actually becomes a motor command, and it is also where VLA designs disagree with each other the most. There are three broad approaches in use right now.

Autoregressive token prediction

You chop each action dimension into discrete bins and predict them one after another, the same next-token setup an LLM already knows how to train. 

OpenVLA uses 256 bins per dimension and a normal next-token loss, so you get to keep the usual LLM tooling. 

What you give up is resolution and speed as the binning leaves jagged motion, and sampling token by token is too slow for anything fast or contact-heavy.

Diffusion and flow-matching decoding

Instead of predicting one token at a time, the model treats action generation as a denoising process and produces a whole chunk of continuous actions at once. 

pi0 was the first VLA built around a flow-matching action head, where they denoised over roughly 10 steps to produce a chunk of up to 50 future actions. 

This made the resulting motion noticeably smoother than anything token-based.

Hybrid two-level architectures

A larger model handles high-level reasoning and breaks the task into steps, whereas a smaller, faster controller handles the precise, high-frequency execution. 

A really good example I like to give for this is Gemini Robotics 1.5, where they paired a reasoning model with an action model.

In my honest opinion, though, after reading through a number of these papers, people spend far more energy arguing about backbone size than about the action head, and I think that is backward. 

If your decoder is squashing a 7-dimensional continuous action into 256 discrete bins, no amount of backbone quality gets that smoothness back!

Key VLA Models in 2026

Now I would like to present a table showing some really important VLA Models which you are likely to run into:

Now lets walk through each one below, starting with the research baseline most people still compare against.

OpenVLA

OpenVLA is a 7-billion-parameter model trained on 970,000 real-world robot demonstrations pulled from Open X-Embodiment. 

It combines a Llama 2 7B backbone with the fused DINOv2 and SigLIP vision encoder, which I described earlier, and the whole model is open source.

The result that was surprising was that it beat the closed RT-2-X model by 16.5 percentage points of task success across 29 tasks, whilst RT-2-X running 55 billion parameters against OpenVLA’s 7 billion! 

Total training took 64 A100 GPUs for 15 days.

The reason it is still popular is that LoRA fine-tuning and 4-bit quantization both work well on it. 

This means a 24GB RTX 4090 can run a quantized version and most of the published work on VLA attacks, defenses and pruning uses OpenVLA as the target simply because its so accessible. 

A follow-up - OpenVLA-OFT improved the fine-tuning recipe for both speed and success rate.

Octo

Octo is a small VLA, and it comes in two sizes - 27 million and 93 million parameters on ViT-S and ViT-B backbones, respectively, and can be run on a single RTX 4090 at 20 to 30 Hz.

It was trained on roughly 800,000 trajectories from Open X-Embodiment across 22 robot embodiments, and its transformer architecture is genuinely modular. 

You can add a new observation input or swap the action head during fine-tuning without rebuilding the model from scratch, and it supports discrete tokens and diffusion-based continuous actions depending on what you need.

Octo’s transformer: task and observation tokens in, readout tokens out to the action head, with room to add new sensors or action spaces at finetuning. Source: Octo paper.

I personally use Octo whenever the limiting factor is compute rather than task difficulty. 

It also supports goal-image conditioning, which is a nice option when describing your target state in words is more awkward than just showing a photo of what you want.

pi0 and the Physical Intelligence family

pi0 from Physical Intelligence is really good at dexterity. It starts from PaliGemma, a 3-billion-parameter open VLM, and adds a 300-million-parameter action expert trained from scratch, giving 3.3 billion parameters total. 

It uses flow matching to produce a chunk of up to 50 future actions at once rather than one step at a time.

The key result was the bimanual tasks such as folding laundry, packing groceries, and assembling boxes, all handled with a level of dexterity nothing before it had shown.

pi0 robot folding laundry using two arms. This is one of the bimanual tasks I mentioned, laundry folding, and it’s worth watching rather than just reading about. Source: Physical Intelligence, “pi0: Our First Generalist Policy”.

Flow matching is doing the real work here, because coordinating two arms at once needs smooth, continuous 16-dimensional output and neither discrete tokens nor slow diffusion sampling can give us that at a usable speed.

The pi0 family has kept on evolving:

• pi0.5 (April 2025): added open-world generalization with a two-stage process that predicts a high-level subtask in words before predicting the low-level action
• pi0.6 (November 2025): learns from the robot’s own experience rather than only from demonstrations
• openpi: checkpoints and an approximate training pipeline are in the openpi repository, pretrained on more than 10,000 hours of robot data

One caveat I do want to state is that pi0’s strongest results depend on proprietary data collected across their own fleet of robots. 

Nobody outside the company has reproduced those exact numbers using only public data, so how much of the performance comes from the architecture versus the data advantage is still genuinely unresolved.

Gemini Robotics 1.5

Gemini Robotics 1.5 from Google DeepMind is best understood as two models working together rather than one. 

Gemini Robotics-ER 1.5 is an embodied reasoning model that plans, calls tools such as Google Search, and can estimate its own progress on a task. 

Gemini Robotics 1.5 is the VLA that carries out what the reasoning model decides.

Gemini Robotics-ER 1.5 on top for embodied reasoning, Gemini Robotics 1.5 underneath as the VLA, shared tool calling in the middle, and actions running on ALOHA 2, Franka, and Apollo without action-space alignment. Source: Gemini Robotics 1.5 paper.

The useful bit is that it mixes actions with an internal reasoning process written in natural language, so the robot thinks before acting and can explain what it decided afterward. 

Given an instruction like “sort my laundry by color”, it first works out that whites go in one bin and everything else in another, then plans the individual picks.

DeepMind reports roughly 250 milliseconds from observation to action, an effective control frequency of 50Hz, achieved by putting the heavy reasoning in the cloud and keeping a lightweight action decoder local to the robot.

The other result I want to state is Motion Transfer, which is a task learned on the dual-arm ALOHA 2 platform carried over directly to a Franka bi-arm setup and to Apptronik’s Apollo humanoid, without retraining on either. 

The weights are closed, so I would say to treat this one as the proprietary frontier baseline rather than something you can download and fine-tune yourself.

SmolVLA

SmolVLA from Hugging Face is 450 million parameters (under an Apache 2.0 license). It pairs a modified SmolVLM2 backbone with a flow-matching action expert, using layer skipping and a reduced number of visual tokens to keep inference cheap enough for small hardware.

It runs on a single consumer GPU, an RTX 3060 included, on a MacBook’s CPU (although it is slow), and it deploys to a Jetson Orin for onboard inference. 

Fine-tuning for 20,000 steps takes around 4 to 5 hours on a single A100 GPU. 

Despite training on fewer than 30,000 episodes, it matches or beats considerably larger models on LIBERO, Meta-World, and real SO-100 and SO-101 arms.

The best part is that the whole thing is pretrained on community-contributed datasets through the LeRobot framework.

How to Choose a VLA for Your Project

Now that I have presented some VLA Models, let's talk about which one to pick for your specific project.

Picking between these five is mostly about matching control frequency and hardware to your actual setup, not chasing whichever one topped a leaderboard last month. 

Here is roughly how I would think through it:

Five constraints, five models. If you only remember one diagram from this article, make it this one. Image by Author.

If you have a consumer GPU or an SO-101 arm sitting on your desk, start with SmolVLA. Nothing else in the open ecosystem gets you a working policy this cheaply.

• If the task needs contact-rich or bimanual manipulation, go straight to pi0 or pi0.5. The flow-matching head is perfect for such dexterity tasks.
• If you are writing a paper and need a baseline that reviewers will recognize, use OpenVLA simply because everyone already has it.
• If compute is the constraint, but you still need multi-embodiment support, Octo is the sensible choice.
• If you want to see what is achievable at the frontier, look at Gemini Robotics 1.5, with the understanding that you cannot self-host it.

And the last major point I would like to make is that a hundred clean demonstrations on SmolVLA will beat fifty messy ones on OpenVLA nearly every time. 

By this, I mean spend 60 to 70 percent of your project time on data collection and quality control, and the remaining on the training.

Applications of VLA Models

VLAs have moved out of demo videos and into a handful of real deployments, though unevenly. 

The pattern I notice is that they land first wherever the environment has too much variation to script, but a human can still step in when something goes wrong. Here are some examples:

Robotic manipulation

This is the primary ground, with tasks such as pick-and-place, kitting, packing, sorting, assembly, etc. 

These are tasks where the robot has to identify, grasp, and reposition objects whose position and orientation change between attempts, which is exactly the condition that scripted systems handle badly.

Warehouse and fulfillment work is the clearest commercial case right now. A scripted picker needs the item presented the same way every time, while a VLA-based one can work through a tote with forty different SKUs jumbled together.

Humanoid robots

Companies including Figure, 1X, and Apptronik are building on VLA foundations, and the reason is simple: a bipedal robot with two working hands has too many possible tasks to script one by one. 

The Gemini Robotics Motion Transfer result, where a skill trained on ALOHA 2 carried straight over to Apptronik’s Apollo humanoid, is exactly the property a humanoid program needs if it wants to scale.

The same skill, trained once on ALOHA 2, running on Apptronik’s Apollo humanoid with no retraining. Source: Google DeepMind, “Gemini Robotics 1.5 brings AI agents into the physical world”.

Autonomous vehicles

Driving stacks are starting to adopt VLA-style architectures to connect what the car sees with what it should do next, inside one reasoning loop rather than a perception-to-planner handoff. 

The appeal is the same as in manipulation: rare situations that no rule ever anticipated, handled by a model that has seen enough of the world to reason about them instead of freezing.

Surgical and medical robotics

I would like to mention that this is the earliest-stage application on this list. 

The idea is that a system able to follow a surgeon’s spoken instruction and adjust to variable tissue conditions beats a rigidly programmed one, but regulatory approval is the main constraint here, not the model's capability. 

I would expect assistive and training applications long before anything close to autonomous.

Vision Language Action Models: Challenges and Limitations

Now, whilst VLAs are great, they still have weaknesses, particularly in four specific places:

Mid-task recovery

VLAs handle variation at the start of a task reasonably well, but when a grasp slips halfway through, or an object shifts under the gripper, recovery is inconsistent. 

Most demonstration data captures successful attempts, so the model has seen very few examples of what recovering from a mistake actually looks like.

On-device inference

Fine manipulation needs a control loop running at 10 to 100Hz, and getting a model of 7 billion or more parameters to hit that rate on the robot’s own hardware is still an aspiration rather than solved engineering. 

This is why most deployed systems either tether to an edge server or split reasoning from control, the way Gemini Robotics 1.5 does. 

Quantization, layer skipping, and distillation all help, but they are workarounds people are still iterating on rather than settled answers.

The data bottleneck

Now that capable open weights exist for anyone to download, data has become the real differentiator. 

Collecting diverse, well-annotated teleoperation data is slow and expensive in a way that scraping the web never was, and the gap between pi0’s published results and what anyone has reproduced with public data alone is the clearest evidence of what a private data advantage actually buys you.

Generalization limits

Current VLAs generalize well inside their training distribution and degrade noticeably outside it. 

A model trained across 22 embodiments on tabletop manipulation can still do poorly on a task shape it has genuinely never seen.

Final Thoughts

As you can see, VLA models bring perception, language understanding, and action generation into one architecture, and that is a structural shift rather than an incremental one. 

What they can do that classical control never managed is generalize to conditions nobody explicitly trained them for, because the language and vision backbones already carry a working model of the physical world into the robot before it ever sees a demonstration.

What really surprises me most about where things stand in 2026 is how much the hard question has moved. 

Three years ago, the hard question was architecture. 

Now that OpenVLA, Octo, pi0, and SmolVLA are all sitting on GitHub for anyone to download, the hard questions are data quality, recovery behavior, and whether you can actually run any of this at 50Hz on the robot itself rather than in a data center.

If you want to keep going, DataCamp’s Introduction to AI Agents course covers the broader agentic context, and Large Language Models (LLMs) Concepts covers the backbone side of things.