Atlasof World Models
Start

Lesson 7 of 7 Joint embedding world models, from the ground up 2 min

V-JEPA 2 and planning with a world model

V-JEPA 2 adds actions to a video world model and plans robot moves by searching for actions whose predicted outcome matches a goal image.

A white robot arm on a pale wooden table beside a cobalt cup and an empty plate, an observatory dome in the window behind.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

Sonu, imagine you want to put your library book back on the top shelf. Before you move, close your eyes for a moment.

In your head, try a few ways. Reach straight up, stand on the stool first or use both hands.

For each way, you picture how it ends. Book on the shelf, or book on the floor.

Then you pick the way whose ending looks most like the goal. That is planning. You tested moves in your head, where mistakes are free.

You also do something clever. You take only the first step. Then you open your eyes, look again and plan the next step. If the stool wobbles, you change your plan.

Scientists built a robot arm that plans this way. First its model watched a giant pile of everyday videos. It learned how things usually move.

Then it watched a smaller pile of robot videos. These showed the arm moving and what happened next. From those, it learned what its own moves do.

Now you show the robot a photo of the goal. Say, a cup sitting on a plate.

The robot imagines many short moves in its head. It keeps the move whose imagined result looks most like the photo. It makes that one move. Then it looks and imagines again.

The surprising part is where it worked. The robot had never practised in those rooms before. It still picked up cups and placed them a good share of the time.

It is not magic yet. It needs a photo of the goal. Words do not work yet. It can only plan a few moves ahead before its guesses drift.

Try this at home. Pick a small job, like moving a toy into a box. Close your eyes and imagine three ways to do it. Pick one and do only the first step.

Did you need to change the plan after you looked?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

V-JEPA 2 (Assran et al., 2025) scales the video recipe, then adds actions. It reports understanding, anticipation and robot planning from one pretrained encoder.

Pretraining comes first. The encoder learned from over 1 million hours of internet video and images. The dataset, VideoMix22M, holds about 22 million samples. The encoder grew to a ViT-g of about 1 billion parameters.

Training ran up to 252,000 iterations. Resolution and clip length rose only during a cooldown phase. The paper credits that schedule with an 8.4 times cut in GPU time.

Reported results without actions:

  • 77.3 top-1 on Something-Something-v2, a motion benchmark
  • 39.7 recall at 5 on Epic-Kitchens-100 action anticipation
  • 84.0 on PerceptionTest and 76.9 on TempCompass, with an 8 billion parameter language model

Then come actions. V-JEPA 2-AC freezes the encoder and trains a new predictor of about 300 million parameters. It has 24 layers and block-causal attention. Each frame's tokens attend only to the present and the past.

Actions and arm states are both 7-dimensional. Training used under 62 hours of unlabeled Droid robot video. Unlabeled here means no rewards and no success flags.

Two losses are summed. Teacher forcing predicts one step ahead from real features. A two-step rollout feeds the model its own prediction once.

Planning is search in representation space. Encode the current frame and a goal image. Choose actions that bring the predicted representation closest to the goal's.

a* = argmin over actions of || P(E(xₜ), a) − E(x_goal) ||₁

The search uses the cross-entropy method. Sample 800 action sequences from a Gaussian. Keep the best, refit and repeat ten times. Execute the first action, observe, then plan again.

On one RTX 4090, each action took 16 seconds. A pixel video model, Cosmos, took 4 minutes per action in the same setup.

Zero-shot results come from two labs that are not in Droid. Each lab ran a Franka arm with 10 trials per task. Averages across both labs:

  • reach, 100 percent for V-JEPA 2-AC and for Octo
  • grasp a cup, 65 percent against Octo's 15
  • pick and place a cup, 80 percent against 15
  • pick and place a box, 65 percent against 10

The authors list clear limits.

  • Planning is sensitive to camera position. Arm axes come from an uncalibrated camera.
  • Long horizons compound error. The action search also grows exponentially with horizon.
  • Goals must be images. Language goals are left for future work.

The utility analogy is switching. A goal state is a feeder configuration. Actions are switch operations. Short horizons and replanning after each step fit that work.

What I would measure in any pilot:

  • success rate with confidence intervals, since 10 trials is a small sample
  • planning time per action against the control deadline
  • error growth by horizon step, open loop and with replanning

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

This lesson is where world models start to act. That changes the risk picture more than any earlier lesson.

V-JEPA 2 shows a pattern worth watching. Learn general physics from public video. Then learn your own machine from a small amount of its own footage. Under 62 hours of robot video was enough for the reported results.

For us, the parallel is our own operations history. Picture a general model of how systems behave. It would be tuned on a modest log of our actions and outcomes.

The reported results are strong for a first attempt. A robot arm picked and placed a cup 80 percent of the time. It had never seen those labs. A widely used baseline policy managed 15 percent.

I would read those numbers with care.

  • Each task had 10 trials per lab.
  • Results came from two labs.
  • The setup was sensitive to camera position.

Planning cost is now a budget line. Each action took 16 seconds of search on one GPU. A pixel-based rival took 4 minutes.

For a robot arm, 16 seconds is slow but workable. For grid switching it may fit. For protection, which acts in fractions of a second, it does not.

The governance question is new too. A planner chooses actions by imagining outcomes. If its imagination drifts, it acts with confidence on a wrong picture. The authors name long-horizon error as a known limit.

The decision is where to try planning first. I would choose a short-horizon task with a clear goal state. A human would approve each step. Field switching plans are one candidate.

Who would sign off on each planned action during the pilot?

Quick check

3 questions. Pick one answer each. Get them all right to mark this lesson as checked.

1In V-JEPA 2-AC, what does the planner try to minimise?

2How much robot video was used to train the action-conditioned predictor?

3Which limit do the authors name for long horizons?

Key terms

  • action-conditioned predictor
  • block-causal attention
  • teacher forcing
  • rollout loss
  • goal image
  • energy in representation space
  • cross-entropy method
  • receding horizon
  • zero-shot transfer
  • compounding error

Further reading

  1. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning Mido Assran, Adrien Bardes, David Fan, Quentin Garrido and others, 2025 Pretraining at scale, the action-conditioned predictor, planning and the zero-shot robot results.
  2. Revisiting Feature Prediction for Learning Visual Representations from Video Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, Nicolas Ballas, 2024 The V-JEPA recipe that V-JEPA 2 scales.
  3. A Path Towards Autonomous Machine Intelligence (version 0.9.2) Yann LeCun, 2022 Proposes H-JEPA world models for planning under uncertainty, the agenda V-JEPA 2-AC begins to test.
  4. Intuitive physics understanding emerges from self-supervised pretraining on natural videos Quentin Garrido, Nicolas Ballas, Mahmoud Assran, Adrien Bardes, Laurent Najman, Michael Rabbat, Emmanuel Dupoux, Yann LeCun, 2025 Evidence that the pretrained video representation holds some physics before any actions are added.

Search the atlas