T-12 Province IV · The Moving Image The Long Gallery 5 min 12 of 21 New

Video World Models II: Interaction, Memory & Efficiency

Keeping a generated world consistent while someone moves around inside it, and doing it fast enough to feel live.

A long white corridor with arched windows in morning mist, one cobalt door and a telescope far down the hall.

In one sentence A playable world must remember what is behind you, respond instantly, and not slowly melt.

Sonu, 10

Imagine a video game where nobody built the levels. Every time you press a button, a computer paints the next moment on the spot. Walk forward and it paints a hallway. Turn left and it paints a door.

That sounds like magic. It is also hard to do well.

Think about your own room. If you turn around and turn back, your bed is still there. Your posters have not moved. The computer has to remember that too.

If it forgets, you turn back and your bed has become a bookshelf. A game like that would feel like a bad dream.

There is a second problem. Each new picture is built from the one before it. A tiny mistake in one picture gets copied into the next one. Then it gets copied again.

After ten minutes the room can smear like a chalk drawing in the rain. The computer did not mean to. Small mistakes just piled up.

The third problem is speed. A game that takes a whole second to answer your button feels broken. One research game showed 20 new pictures every second. That is fast enough to play.

Researchers work on all three problems at once. Some give the computer a kind of notebook for places it has seen. Some let it practise on its own small mistakes. Then it learns to fix them before they grow.

Try this at home. Walk into a room you do not use much. Look around for ten seconds, then walk out. Draw the room from memory.

Now go back and compare. What did your memory keep? What did it change?

Senior AI Architect

Genie (Bruce et al., 2024) learned a latent action space from unlabelled internet video. It inferred which controls exist without any action labels. A user can then drive those actions frame by frame.

Genie has 11 billion parameters. It pairs a video tokenizer with an autoregressive dynamics model. A small latent action model sits beside them.

Interactive systems generate autoregressively in a closed loop. Small errors feed back as inputs and accumulate. This is closed-loop drift. It is the main enemy of long sessions.

Three mitigations recur. The first is noise augmentation on context frames during training. The model learns to recover from its own mistakes.

The second is Diffusion Forcing (Chen et al., 2024). It trains with independent noise levels per token. Its rollouts run past the training horizon where baselines diverge. The third is periodic re-anchoring to a known frame or map.

GameNGen (Valevski et al., 2024) shows how far this goes. It simulated DOOM at 20 frames per second on a single TPU. Conditioning augmentations kept multi-minute play stable. Human raters were only slightly better than chance at spotting the fake clips.

Memory is bounded by the context window. One approach compresses long-term memory into a state. Another retrieves earlier frames by pose or location. Explicit 3D scaffolds keep geometry stable across revisits.

Genie 3 (Google DeepMind, 2025) runs at 24 frames per second in 720p. DeepMind reports visual memory that reaches back about a minute. Its worlds stay consistent for a few minutes.

Efficiency work targets frame rates a person can use. The levers are few-step samplers, smaller latents and causal attention caches. Each one trades some fidelity for speed. The right trade depends on whether a person or a policy is in the loop.

A policy can wait a little longer for each frame. A person at a console cannot.

Take a control-room training simulator built on such a model. A trainee opens a breaker in a generated substation. She walks away and returns. The breaker must still read open.

A model that forgets here teaches the wrong state. That is worse than having no simulator at all.

Test revisits directly. Script a route that returns to the same bay after longer and longer delays. Score whether the state on return matches the state on departure. Plot that score against the delay.

Report these together:

Revisit consistency on scripted routes. - Drift per minute against a reference rollout. - Latency from input to displayed frame.

A fast model that forgets fails the training use case. A faithful model at two frames a second fails it too.

My CIO

Interactive video worlds open real applications. They can produce training data for robots and cars. They can prototype environments in hours. They also allow new kinds of media.

For a utility, the nearest use is training. A generated substation or storm scene could let an operator rehearse rare events. Those events are hard to stage for real.

The readiness question is duration and fidelity under interaction. A demo runs for thirty seconds on a path someone chose. A product runs for an hour on paths nobody chose.

Three numbers separate a demo from a product:

How long a session stays coherent. - What happens when a user revisits a place. - The cost per minute of interactive generation.

The first two are about trust. A training simulator that forgets a switch position teaches the wrong habit. That is a safety problem as well as a quality one.

The third is about money. Interactive generation runs on accelerators for every user and every minute. Cost scales with use.

A pilot with ten trainees can look cheap. A rollout to every control room can look very different.

The decision is whether to pilot now or wait. I would recommend to pilot on one narrow, scripted scenario. Build the revisit test in from the first day. That gives real numbers on all three without a large commitment.

Waiting has a cost too. The models are improving every few months. A team that has already run a scripted pilot will know what to ask for.

Ask the vendor one question. What is the cost per coherent minute, measured on revisits?

Quick check

Key terms