-
Mission · Sonu, 10
V-JEPA 2 and planning with a world model
V-JEPA 2 adds actions to a video world model and plans robot moves by searching for actions whose predicted outcome matches a goal image.
5 cards, 1 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 5
Imagine, compare, act
A planning world model imagines the result of each candidate action. It keeps the action whose imagined result is closest to the goal.
-
Try this · 2 of 5
Plan with eyes closed
Pick a small job, like moving a toy into a box. Imagine three ways. Do only the first step. Look and plan again.
-
Quick check · 3 of 5
Quick check on results
How often did the arm pick and place a cup in unseen labs?
Say your answer, then show it
80 percent, averaged over two labs with 10 trials each.
-
Idea · 4 of 5
Known limits
Planning depends on camera position. Errors compound over long horizons. Goals must be images for now. The authors state all three.
-
Recap · 5 of 5
What to carry forward
Pretrain on video and add a small action predictor. Plan by searching for actions that match a goal image. Short horizons and replanning keep error in check.
-
Mission · Senior AI Architect
V-JEPA 2 and planning with a world model
V-JEPA 2 adds actions to a video world model and plans robot moves by searching for actions whose predicted outcome matches a goal image.
10 cards, 2 of them quick checks. About 5 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 10
Imagine, compare, act
A planning world model imagines the result of each candidate action. It keeps the action whose imagined result is closest to the goal.
-
Idea · 2 of 10
Two kinds of data
V-JEPA 2 learned general motion from over a million hours of video. Its action predictor then learned the arm from under 62 hours of robot footage.
-
Idea · 3 of 10
The action predictor
A 300 million parameter transformer reads frozen video features plus 7-dimensional actions and arm states. Block-causal attention keeps the future hidden from the past.
-
Try this · 4 of 10
Write the energy
Write the planning score in words. It is the distance between the predicted summary and the goal summary. Mark what you would change to lower it.
-
Quick check · 5 of 10
Quick check on planning
What does V-JEPA 2-AC minimise when planning?
Say your answer, then show it
The L1 distance between predicted and goal representations.
-
Idea · 6 of 10
Search by sampling
The cross-entropy method samples 800 action sequences, keeps the best and refits, ten times. The robot runs the first action and plans again.
-
Quick check · 7 of 10
Quick check on results
How often did the arm pick and place a cup in unseen labs?
Say your answer, then show it
80 percent, averaged over two labs with 10 trials each.
-
Idea · 8 of 10
Known limits
Planning depends on camera position. Errors compound over long horizons. Goals must be images for now. The authors state all three.
-
Try this · 9 of 10
Measure drift
Take any forecast you own. Compare its error one step ahead with its error twenty steps ahead. Write both numbers down.
-
Recap · 10 of 10
What to carry forward
Pretrain on video and add a small action predictor. Plan by searching for actions that match a goal image. Short horizons and replanning keep error in check.
-
Mission · My CIO
V-JEPA 2 and planning with a world model
V-JEPA 2 adds actions to a video world model and plans robot moves by searching for actions whose predicted outcome matches a goal image.
6 cards, 2 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea · 1 of 6
Imagine, compare, act
A planning world model imagines the result of each candidate action. It keeps the action whose imagined result is closest to the goal.
-
Try this · 2 of 6
Price the search
Each action took 16 seconds on one GPU. Pick one task at your work. Is 16 seconds per step inside its deadline?
-
Quick check · 3 of 6
Quick check on results
How often did the arm pick and place a cup in unseen labs?
Say your answer, then show it
80 percent, averaged over two labs with 10 trials each.
-
Idea · 4 of 6
Known limits
Planning depends on camera position. Errors compound over long horizons. Goals must be images for now. The authors state all three.
-
Quick check · 5 of 6
Quick check on risk
Why keep a human approving each planned step at first?
Say your answer, then show it
An imagined outcome can drift. The planner then acts on it with confidence.
-
Recap · 6 of 6
What to carry forward
Pretrain on video and add a small action predictor. Plan by searching for actions that match a goal image. Short horizons and replanning keep error in check.