Atlasof World Models
Start

T-12 · Video World Models II: Interaction, Memory & Efficiency

  1. Mission · Sonu, 10

    Video World Models II: Interaction, Memory & Efficiency

    A playable world must remember what is behind you, respond instantly, and not slowly melt.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Imagine a video game where nobody built the levels. Every time you press a button, a computer paints the next moment on the spot. Walk forward and it paints a hallway. Turn left and it paints a door.

    That sounds like magic. It is also hard to do well.

  3. Idea 2 of 4

    Keep in your head Imagine a video game where nobody built the levels.

    Think about your own room. If you turn around and turn back, your bed is still there. Your posters have not moved. The computer has to remember that too.

    If it forgets, you turn back and your bed has become a bookshelf. A game like that would feel like a bad dream.

    Try this Close the screen. Tell someone at home what this card says, in your own words.

  4. Idea 3 of 4

    Keep in your head Think about your own room.

    There is a second problem. Each new picture is built from the one before it. A tiny mistake in one picture gets copied into the next one. Then it gets copied again.

    After ten minutes the room can smear like a chalk drawing in the rain. The computer did not mean to. Small mistakes just piled up.

  5. Idea 4 of 4

    Keep in your head There is a second problem.

    The third problem is speed. A game that takes a whole second to answer your button feels broken. One research game showed 20 new pictures every second. That is fast enough to play.

    Researchers work on all three problems at once. Some give the computer a kind of notebook for places it has seen. Some let it practise on its own small mistakes. Then it learns to fix them before they grow.

    Try this at home. Walk into a room you do not use much. Look around for ten seconds, then walk out. Draw the room from memory.

    Now go back and compare. What did your memory keep? What did it change?

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1What did Genie learn without any action labels?

  8. Quick check 2 of 3

    2Why do interactive video models drift over a long session?

  9. Quick check 3 of 3

    3Which test checks a playable world's memory most directly?

  10. Recap

    What you just learned

    1. Imagine a video game where nobody built the levels.
    2. Think about your own room.
    3. There is a second problem.
    4. The third problem is speed.

  1. Mission · Senior AI Architect

    Video World Models II: Interaction, Memory & Efficiency

    A playable world must remember what is behind you, respond instantly, and not slowly melt.

    7 idea cards, 3 questions and a recap. About 7 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 7

    Genie (Bruce et al., 2024) learned a latent action space from unlabelled internet video. It inferred which controls exist without any action labels. A user can then drive those actions frame by frame.

    Genie has 11 billion parameters. It pairs a video tokenizer with an autoregressive dynamics model. A small latent action model sits beside them.

  3. Idea 2 of 7

    Keep in your head Genie (Bruce et al., 2024) learned a latent action space from unlabelled internet video.

    Interactive systems generate autoregressively in a closed loop. Small errors feed back as inputs and accumulate. This is closed-loop drift. It is the main enemy of long sessions.

    Three mitigations recur. The first is noise augmentation on context frames during training. The model learns to recover from its own mistakes.

    Try this Find "closed-loop drift" on this card. Define it in one line without looking.

  4. Idea 3 of 7

    Keep in your head Interactive systems generate autoregressively in a closed loop.

    The second is Diffusion Forcing (Chen et al., 2024). It trains with independent noise levels per token. Its rollouts run past the training horizon where baselines diverge. The third is periodic re-anchoring to a known frame or map.

    GameNGen (Valevski et al., 2024) shows how far this goes. It simulated DOOM at 20 frames per second on a single TPU. Conditioning augmentations kept multi-minute play stable. Human raters were only slightly better than chance at spotting the fake clips.

  5. Idea 4 of 7

    Keep in your head The second is Diffusion Forcing (Chen et al., 2024).

    Memory is bounded by the context window. One approach compresses long-term memory into a state. Another retrieves earlier frames by pose or location. Explicit 3D scaffolds keep geometry stable across revisits.

    Genie 3 (Google DeepMind, 2025) runs at 24 frames per second in 720p. DeepMind reports visual memory that reaches back about a minute. Its worlds stay consistent for a few minutes.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 7

    Keep in your head Memory is bounded by the context window.

    Efficiency work targets frame rates a person can use. The levers are few-step samplers, smaller latents and causal attention caches. Each one trades some fidelity for speed. The right trade depends on whether a person or a policy is in the loop.

    A policy can wait a little longer for each frame. A person at a console cannot.

  7. Idea 6 of 7

    Keep in your head Efficiency work targets frame rates a person can use.

    Take a control-room training simulator built on such a model. A trainee opens a breaker in a generated substation. She walks away and returns. The breaker must still read open.

    A model that forgets here teaches the wrong state. That is worse than having no simulator at all.

    Try this Write down the one number on this card you would put on a dashboard.

  8. Idea 7 of 7

    Keep in your head Take a control-room training simulator built on such a model.

    Test revisits directly. Script a route that returns to the same bay after longer and longer delays. Score whether the state on return matches the state on departure. Plot that score against the delay.

    Report these together:

    • Revisit consistency on scripted routes. - Drift per minute against a reference rollout. - Latency from input to displayed frame.

    A fast model that forgets fails the training use case. A faithful model at two frames a second fails it too.

  9. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  10. Quick check 1 of 3

    1What did Genie learn without any action labels?

  11. Quick check 2 of 3

    2Why do interactive video models drift over a long session?

  12. Quick check 3 of 3

    3Which test checks a playable world's memory most directly?

  13. Recap

    What you just learned

    1. Genie (Bruce et al., 2024) learned a latent action space from unlabelled internet video.
    2. Interactive systems generate autoregressively in a closed loop.
    3. The second is Diffusion Forcing (Chen et al., 2024).
    4. Memory is bounded by the context window.
    5. Efficiency work targets frame rates a person can use.
    6. Take a control-room training simulator built on such a model.
    7. Test revisits directly.

  1. Mission · My CIO

    Video World Models II: Interaction, Memory & Efficiency

    A playable world must remember what is behind you, respond instantly, and not slowly melt.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Interactive video worlds open real applications. They can produce training data for robots and cars. They can prototype environments in hours. They also allow new kinds of media.

    For a utility, the nearest use is training. A generated substation or storm scene could let an operator rehearse rare events. Those events are hard to stage for real.

  3. Idea 2 of 4

    Keep in your head Interactive video worlds open real applications.

    The readiness question is duration and fidelity under interaction. A demo runs for thirty seconds on a path someone chose. A product runs for an hour on paths nobody chose.

    Three numbers separate a demo from a product:

    • How long a session stays coherent. - What happens when a user revisits a place. - The cost per minute of interactive generation.

    Try this Find "interactive generation" on this card. Define it in one line without looking.

  4. Idea 3 of 4

    Keep in your head The readiness question is duration and fidelity under interaction.

    The first two are about trust. A training simulator that forgets a switch position teaches the wrong habit. That is a safety problem as well as a quality one.

    The third is about money. Interactive generation runs on accelerators for every user and every minute. Cost scales with use.

  5. Idea 4 of 4

    Keep in your head The first two are about trust.

    A pilot with ten trainees can look cheap. A rollout to every control room can look very different.

    The decision is whether to pilot now or wait. I would recommend to pilot on one narrow, scripted scenario. Build the revisit test in from the first day. That gives real numbers on all three without a large commitment.

    Waiting has a cost too. The models are improving every few months. A team that has already run a scripted pilot will know what to ask for.

    Ask the vendor one question. What is the cost per coherent minute, measured on revisits?

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1What did Genie learn without any action labels?

  8. Quick check 2 of 3

    2Why do interactive video models drift over a long session?

  9. Quick check 3 of 3

    3Which test checks a playable world's memory most directly?

  10. Recap

    What you just learned

    1. Interactive video worlds open real applications.
    2. The readiness question is duration and fidelity under interaction.
    3. The first two are about trust.
    4. A pilot with ten trainees can look cheap.

Search the atlas