T-17 Province V · The Frontier The Embassy 5 min 17 of 21 New

World Models for Robots II: Vision-Language-Action & World-Action Models

Robot policies built on large vision-language models, and the push to give them an explicit sense of consequences.

A white robot arm reaching toward a bowl on a tray, a cobalt cup beside it, on a long bench.

In one sentence Give a robot a model that already understands pictures and words, then teach it to answer in movements.

Sonu, 10

Ask a person to put the apple in the bowl next to the blue cup. They look at the table. They understand the words. Then they move, all in one smooth go.

Now picture a helper robot in our kitchen. We want it to do the same thing. It has to see the table and find the blue cup.

It has to understand what "next to" means. Then it has to move its arm the right way.

The robots in this territory start with a head start. Their brains first learned from millions of pictures and sentences on the internet. So they already know what an apple looks like. They know what a bowl is.

Then they watch people guide real robot arms through many tasks. From that practice, they learn to answer in movements instead of words. Builders call these vision-language-action models. See, understand, act.

There is one problem. Our kitchen robot acts without thinking ahead. It might reach across the table and knock the milk over. It never pictured the milk tipping.

So the newest robots add a little imagination. Before the arm moves, the robot pictures what will happen next. If the picture shows spilled milk, it picks a different path. Builders call these world-action models.

You do this all day without noticing. Before you pour juice, you already see the glass filling up. You stop before it spills because you saw the spill first.

Try this at home. Play robot with a grown-up. Give them one short instruction, like "put the spoon next to the red mug".

Before they move, ask them to say what will happen. Then watch whether their guess comes true.

Senior AI Architect

A vision-language-action model is a vision-language model that answers in actions. RT-2 (Brohan et al., 2023) fine-tuned a VLM to emit discretized action tokens alongside text. Web knowledge transferred to control. Across 6,000 evaluation trials it generalized better to novel objects.

It also followed commands that never appeared in its robot data. One example placed an object on a particular number or icon. That is web knowledge doing robot work.

OpenVLA (Kim et al., 2024) made the recipe open. It is a 7B model trained on 970k real robot demonstrations. It beat the closed 55B RT-2-X by 16.5 points of success across 29 tasks. It did that with about seven times fewer parameters.

π0 (Black et al., 2024) swapped token decoding for a flow-matching action head. It sits on a pretrained VLM and gives smooth, high-frequency continuous control. It trained across single-arm robots, dual-arm robots and mobile manipulators. Its tasks include laundry folding and box assembly.

Pure VLAs are reactive. They map observation and instruction to action without modelling consequences. That is the gap this territory now works on.

World-action models couple action generation to a prediction of how the world will evolve. A 2026 tutorial (Zhang et al., 2026) sorts them into four families:

imagine first, then execute
predict actions conditioned on video features
model video and action jointly
add video prediction as an auxiliary loss for the policy

UniPi (Du et al., 2023) is the clearest early case of the first family. It generates a text-conditioned video of the plan, then extracts actions from the frames.

The policy is now consistent with a predicted future. You can also check it against that future. This mirrors the arc of the whole atlas, from imitation toward explicit prediction.

The trade-offs are concrete. Token heads are simple but coarse. Flow heads are smooth but harder to inspect.

Imagined video costs compute at every control step. Long-horizon reliability remains the open problem.

The failure modes repeat across labs. A policy can succeed on the demo object and fail on a new one. It can execute a valid motion in a state that changed a second ago. It almost never knows that it is about to fail.

Take a meter shop that tests returned meters. A VLA gets one instruction. Move every meter with a cracked cover to the reject rack.

The words are easy for a VLM. The hard part is a meter model it never saw, sitting at an odd angle. A world-action model should predict the grasp before it commits.

I would recommend logging that prediction next to the outcome.

Measure success over hundreds of trials. A short video proves little.

Split results by seen and unseen objects. Track how often the predicted next frame matched what happened. Count the failures it flagged in advance.

My CIO

Generalist robot policies are improving fast. They borrow scale from models trained on the internet. A robot that already knows what a cup is learns new tasks from fewer demonstrations.

The weak point is reliability. A video of a robot folding laundry shows one success. It does not show the many tries that failed. Long, multi-step tasks fail far more often than short ones.

For a utility, the near-term uses are narrow. Think of inspection in a substation yard or sorting in a meter shop. Each one has a clear task, a fixed site and a safety rule. That is where a generalist policy can earn its keep first.

The money question is buy versus build. Few utilities should train these models. Most will buy a system and fine-tune it on site data.

The cost then sits in data collection and site integration. It also sits in the safety case.

Ask vendors for evidence in this order:

success rates across many trials, split by seen and unseen objects
how performance falls as tasks get longer
whether the system flags its own likely failures before acting

That last point separates a demo from a deployment. The newer world-action models predict what their actions will do. That prediction gives you something to check and log. A reactive policy gives you nothing until after the arm has moved.

I would fund pilots only where the vendor reports trial counts openly. A demo reel is marketing. A trial log is evidence.

The one question to ask the vendor is short. Out of a hundred tries on objects it has never seen, how many succeed?

Quick check

Key terms