Atlasof World Models
Start

Lesson 6 of 7 · V-JEPA, learning from video

  1. Mission · Sonu, 10

    V-JEPA, learning from video

    V-JEPA hides large tubes of space and time in a video and predicts their summaries. The result shows signs of intuitive physics.

    6 cards, 2 of them quick checks. About 3 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea · 1 of 6

    Hide a tube

    V-JEPA hides a block in every frame of a clip. The hidden region becomes a tube through time. The model predicts the tube's summary.

  3. Try this · 2 of 6

    The cushion test

    Roll a ball behind a cushion while someone watches. Ask where it will come out. Stop it secretly. Watch their face.

  4. Quick check · 3 of 6

    Quick check on surprise

    How did the physics study measure surprise?

    Say your answer, then show it

    As the error between predicted and observed representations. Impossible clips should score higher.

  5. Idea · 4 of 6

    Signs of intuitive physics

    V-JEPA reached 98 percent on IntPhys pairs. It beat untrained networks on object permanence and continuity. Pixel models and two language models stayed near chance.

  6. Quick check · 5 of 6

    Quick check on limits

    Where did V-JEPA fail the physics tests?

    Say your answer, then show it

    Collision and solidity on GRASP. Gravity and solidity on InfLevel-lab.

  7. Recap · 6 of 6

    What to carry forward

    Hide tubes, predict their summaries with a slow target and probe the frozen encoder. Surprise in representation space tracks some physics. Collisions remain weak.

    Back to the full lesson Next mission: V-JEPA 2 and planning with a world model

  1. Mission · Senior AI Architect

    V-JEPA, learning from video

    V-JEPA hides large tubes of space and time in a video and predicts their summaries. The result shows signs of intuitive physics.

    10 cards, 3 of them quick checks. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea · 1 of 10

    Hide a tube

    V-JEPA hides a block in every frame of a clip. The hidden region becomes a tube through time. The model predicts the tube's summary.

  3. Idea · 2 of 10

    Why tubes

    Neighbouring frames look almost the same. A patch hidden in one frame can be copied from the next. Hiding the same block in every frame removes that shortcut.

  4. Idea · 3 of 10

    Two mask sizes

    Short-range masks join 8 small blocks. Long-range masks join 2 large ones. Together they hide about 90 percent of each clip.

  5. Try this · 4 of 10

    Count the tokens

    A clip has 16 frames of 224 by 224 pixels. Each token spans 2 frames and 16 by 16 pixels. How many tokens? Check against 1,568.

  6. Quick check · 5 of 10

    Quick check on masking

    Why mask the same block in every frame?

    Say your answer, then show it

    So the model cannot copy the hidden patch from a neighbouring frame.

  7. Idea · 6 of 10

    Frozen and probed

    V-JEPA is judged with its encoder frozen. A small attentive probe reads the features. The ViT-H/16 reached 81.9 percent on Kinetics-400.

  8. Quick check · 7 of 10

    Quick check on surprise

    How did the physics study measure surprise?

    Say your answer, then show it

    As the error between predicted and observed representations. Impossible clips should score higher.

  9. Idea · 8 of 10

    Signs of intuitive physics

    V-JEPA reached 98 percent on IntPhys pairs. It beat untrained networks on object permanence and continuity. Pixel models and two language models stayed near chance.

  10. Quick check · 9 of 10

    Quick check on limits

    Where did V-JEPA fail the physics tests?

    Say your answer, then show it

    Collision and solidity on GRASP. Gravity and solidity on InfLevel-lab.

  11. Recap · 10 of 10

    What to carry forward

    Hide tubes, predict their summaries with a slow target and probe the frozen encoder. Surprise in representation space tracks some physics. Collisions remain weak.

    Back to the full lesson Next mission: V-JEPA 2 and planning with a world model

  1. Mission · My CIO

    V-JEPA, learning from video

    V-JEPA hides large tubes of space and time in a video and predicts their summaries. The result shows signs of intuitive physics.

    7 cards, 2 of them quick checks. About 4 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea · 1 of 7

    Hide a tube

    V-JEPA hides a block in every frame of a clip. The hidden region becomes a tube through time. The model predicts the tube's summary.

  3. Try this · 2 of 7

    Find your video pile

    List every camera source your team records. Write the hours per month beside each. Mark which ones have any labels at all.

  4. Quick check · 3 of 7

    Quick check on surprise

    How did the physics study measure surprise?

    Say your answer, then show it

    As the error between predicted and observed representations. Impossible clips should score higher.

  5. Idea · 4 of 7

    Signs of intuitive physics

    V-JEPA reached 98 percent on IntPhys pairs. It beat untrained networks on object permanence and continuity. Pixel models and two language models stayed near chance.

  6. Try this · 5 of 7

    Stage an impossible event

    Think of one event at a site that should never happen on camera. Write how you would stage it safely for a test clip.

  7. Quick check · 6 of 7

    Quick check on limits

    Where did V-JEPA fail the physics tests?

    Say your answer, then show it

    Collision and solidity on GRASP. Gravity and solidity on InfLevel-lab.

  8. Recap · 7 of 7

    What to carry forward

    Hide tubes, predict their summaries with a slow target and probe the frozen encoder. Surprise in representation space tracks some physics. Collisions remain weak.

    Back to the full lesson Next mission: V-JEPA 2 and planning with a world model

Search the atlas