Atlasof World Models
Start

T-17 · World Models for Robots II: Vision-Language-Action & World-Action Models

  1. Mission · Sonu, 10

    World Models for Robots II: Vision-Language-Action & World-Action Models

    Give a robot a model that already understands pictures and words, then teach it to answer in movements.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Ask a person to put the apple in the bowl next to the blue cup. They look at the table. They understand the words. Then they move, all in one smooth go.

    Now picture a helper robot in our kitchen. We want it to do the same thing. It has to see the table and find the blue cup.

  3. Idea 2 of 4

    Keep in your head Ask a person to put the apple in the bowl next to the blue cup.

    It has to understand what "next to" means. Then it has to move its arm the right way.

    The robots in this territory start with a head start. Their brains first learned from millions of pictures and sentences on the internet. So they already know what an apple looks like. They know what a bowl is.

    Try this Close the screen. Tell someone at home what this card says, in your own words.

  4. Idea 3 of 4

    Keep in your head It has to understand what "next to" means.

    Then they watch people guide real robot arms through many tasks. From that practice, they learn to answer in movements instead of words. Builders call these vision-language-action models. See, understand, act.

    There is one problem. Our kitchen robot acts without thinking ahead. It might reach across the table and knock the milk over. It never pictured the milk tipping.

  5. Idea 4 of 4

    Keep in your head Then they watch people guide real robot arms through many tasks.

    So the newest robots add a little imagination. Before the arm moves, the robot pictures what will happen next. If the picture shows spilled milk, it picks a different path. Builders call these world-action models.

    You do this all day without noticing. Before you pour juice, you already see the glass filling up. You stop before it spills because you saw the spill first.

    Try this at home. Play robot with a grown-up. Give them one short instruction, like "put the spoon next to the red mug".

    Before they move, ask them to say what will happen. Then watch whether their guess comes true.

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Words to keep

    VLA

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1What did RT-2 add to a vision-language model?

  8. Quick check 2 of 3

    2What makes a world-action model different from a pure VLA?

  9. Quick check 3 of 3

    3Which evidence should a buyer ask for first?

  10. Recap

    What you just learned

    1. Ask a person to put the apple in the bowl next to the blue cup.
    2. It has to understand what "next to" means.
    3. Then they watch people guide real robot arms through many tasks.
    4. So the newest robots add a little imagination.

  1. Mission · Senior AI Architect

    World Models for Robots II: Vision-Language-Action & World-Action Models

    Give a robot a model that already understands pictures and words, then teach it to answer in movements.

    8 idea cards, 3 questions and a recap. About 8 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 8

    A vision-language-action model is a vision-language model that answers in actions. RT-2 (Brohan et al., 2023) fine-tuned a VLM to emit discretized action tokens alongside text. Web knowledge transferred to control. Across 6,000 evaluation trials it generalized better to novel objects.

    It also followed commands that never appeared in its robot data. One example placed an object on a particular number or icon. That is web knowledge doing robot work.

  3. Idea 2 of 8

    Keep in your head A vision-language-action model is a vision-language model that answers in actions.

    OpenVLA (Kim et al., 2024) made the recipe open. It is a 7B model trained on 970k real robot demonstrations. It beat the closed 55B RT-2-X by 16.5 points of success across 29 tasks. It did that with about seven times fewer parameters.

    π0 (Black et al., 2024) swapped token decoding for a flow-matching action head. It sits on a pretrained VLM and gives smooth, high-frequency continuous control. It trained across single-arm robots, dual-arm robots and mobile manipulators. Its tasks include laundry folding and box assembly.

    Try this Find "VLA" on this card. Define it in one line without looking.

  4. Idea 3 of 8

    Keep in your head OpenVLA (Kim et al., 2024) made the recipe open.

    Pure VLAs are reactive. They map observation and instruction to action without modelling consequences. That is the gap this territory now works on.

    World-action models couple action generation to a prediction of how the world will evolve. A 2026 tutorial (Zhang et al., 2026) sorts them into four families:

  5. Idea 4 of 8

    Keep in your head Pure VLAs are reactive.

    • imagine first, then execute
    • predict actions conditioned on video features
    • model video and action jointly
    • add video prediction as an auxiliary loss for the policy

    UniPi (Du et al., 2023) is the clearest early case of the first family. It generates a text-conditioned video of the plan, then extracts actions from the frames.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 8

    Keep in your head imagine first, then execute predict actions conditioned on video features model video and action jointly add video prediction as an auxiliary loss for the policy ...

    The policy is now consistent with a predicted future. You can also check it against that future. This mirrors the arc of the whole atlas, from imitation toward explicit prediction.

    The trade-offs are concrete. Token heads are simple but coarse. Flow heads are smooth but harder to inspect.

  7. Idea 6 of 8

    Keep in your head The policy is now consistent with a predicted future.

    Imagined video costs compute at every control step. Long-horizon reliability remains the open problem.

    The failure modes repeat across labs. A policy can succeed on the demo object and fail on a new one. It can execute a valid motion in a state that changed a second ago. It almost never knows that it is about to fail.

    Try this Write down the one number on this card you would put on a dashboard.

  8. Idea 7 of 8

    Keep in your head Imagined video costs compute at every control step.

    Take a meter shop that tests returned meters. A VLA gets one instruction. Move every meter with a cracked cover to the reject rack.

    The words are easy for a VLM. The hard part is a meter model it never saw, sitting at an odd angle. A world-action model should predict the grasp before it commits.

  9. Idea 8 of 8

    Keep in your head Take a meter shop that tests returned meters.

    I would recommend logging that prediction next to the outcome.

    Measure success over hundreds of trials. A short video proves little.

    Split results by seen and unseen objects. Track how often the predicted next frame matched what happened. Count the failures it flagged in advance.

    Try this Sketch the data flow on this card as three boxes and two arrows.

  10. Words to keep

    VLA

    Try this Cover the list. Say each word and what it means, then check.

  11. Quick check 1 of 3

    1What did RT-2 add to a vision-language model?

  12. Quick check 2 of 3

    2What makes a world-action model different from a pure VLA?

  13. Quick check 3 of 3

    3Which evidence should a buyer ask for first?

  14. Recap

    What you just learned

    1. A vision-language-action model is a vision-language model that answers in actions.
    2. OpenVLA (Kim et al., 2024) made the recipe open.
    3. Pure VLAs are reactive.
    4. imagine first, then execute predict actions conditioned on video features model video and action jointly add video prediction as an auxiliary loss for the policy ...
    5. The policy is now consistent with a predicted future.
    6. Imagined video costs compute at every control step.
    7. Take a meter shop that tests returned meters.
    8. I would recommend logging that prediction next to the outcome.

  1. Mission · My CIO

    World Models for Robots II: Vision-Language-Action & World-Action Models

    Give a robot a model that already understands pictures and words, then teach it to answer in movements.

    5 idea cards, 3 questions and a recap. About 6 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 5

    Generalist robot policies are improving fast. They borrow scale from models trained on the internet. A robot that already knows what a cup is learns new tasks from fewer demonstrations.

    The weak point is reliability. A video of a robot folding laundry shows one success. It does not show the many tries that failed. Long, multi-step tasks fail far more often than short ones.

  3. Idea 2 of 5

    Keep in your head Generalist robot policies are improving fast.

    For a utility, the near-term uses are narrow. Think of inspection in a substation yard or sorting in a meter shop. Each one has a clear task, a fixed site and a safety rule. That is where a generalist policy can earn its keep first.

    Try this Name the one decision this card changes for you.

  4. Idea 3 of 5

    Keep in your head For a utility, the near-term uses are narrow.

    The money question is buy versus build. Few utilities should train these models. Most will buy a system and fine-tune it on site data.

    The cost then sits in data collection and site integration. It also sits in the safety case.

    Ask vendors for evidence in this order:

  5. Idea 4 of 5

    Keep in your head The money question is buy versus build.

    • success rates across many trials, split by seen and unseen objects
    • how performance falls as tasks get longer
    • whether the system flags its own likely failures before acting

    That last point separates a demo from a deployment. The newer world-action models predict what their actions will do. That prediction gives you something to check and log. A reactive policy gives you nothing until after the arm has moved.

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Idea 5 of 5

    Keep in your head success rates across many trials, split by seen and unseen objects how performance falls as tasks get longer whether the system flags its own likely ...

    I would fund pilots only where the vendor reports trial counts openly. A demo reel is marketing. A trial log is evidence.

    The one question to ask the vendor is short. Out of a hundred tries on objects it has never seen, how many succeed?

  7. Words to keep

    VLA

    Try this Cover the list. Say each word and what it means, then check.

  8. Quick check 1 of 3

    1What did RT-2 add to a vision-language model?

  9. Quick check 2 of 3

    2What makes a world-action model different from a pure VLA?

  10. Quick check 3 of 3

    3Which evidence should a buyer ask for first?

  11. Recap

    What you just learned

    1. Generalist robot policies are improving fast.
    2. For a utility, the near-term uses are narrow.
    3. The money question is buy versus build.
    4. success rates across many trials, split by seen and unseen objects how performance falls as tasks get longer whether the system flags its own likely ...
    5. I would fund pilots only where the vendor reports trial counts openly.

Search the atlas