T-11 Province IV · The Moving Image The Projection Hall 5 min 11 of 21 New

Video World Models I: Generation & Rollout

Treating video prediction as world simulation: what a model must learn to continue a scene plausibly, frame after frame.

Action

A pale ball at the edge of a round table, a cobalt line running away across the floor beneath it.

In one sentence If a model can continue any video realistically, it must have learned something about how the world moves.

Sonu, 10

Roll a ball across a table and stop the video halfway. The ball is close to the edge. Ask a friend what happens next. They will say it falls off.

To answer, they used what they already know. Balls roll. Tables end. Things fall.

They might never be able to explain it. They still get it right.

Some computers learn by playing that game over and over. They use a huge number of videos.

They watch the start of a clip. Then they guess the rest, one picture after another. When the guess is wrong, they adjust.

To get good at this game, the computer has to pick up how things move. Balls roll. Cups do not walk through walls. A dog that runs out of the picture can run back in.

Now add a button. Press "left" and the car in the video turns left. Press "stop" and it slows down.

The video becomes a little pretend world you can steer. That is what people mean by a video world model.

Some video computers make beautiful clips and ignore the button. They are like a movie. You can watch it. You cannot change what happens.

A world model has to listen to the button. It also has to keep the room the same as time goes on. A chair should not turn into a table halfway through.

Try this at school. Make a flipbook in the corner of a notebook. On the first ten pages, draw a ball rolling toward a table edge.

Give it to a friend. Ask them to draw the next five pages. Did they make it fall? Did it fall the way a real ball falls?

Senior AI Architect

Three state choices dominate. Pixel states are faithful and expensive. Token states come from a learned video tokenizer. An autoregressive transformer then predicts the next frame's tokens.

Latent states let diffusion or flow models generate chunks of frames in a compressed space. Most current systems generate in latents and decode to pixels.

GAIA-1 (Hu et al., 2023) is a token model. It maps video, text and action inputs to discrete tokens. It then predicts the next token in the sequence.

A video model acts as a world model only under two conditions. Generation must be causal, so each frame depends only on the past. It must also be action-conditioned.

A bidirectional clip generator can look beautiful. It still cannot be stepped forward one action at a time. Every frame in it can see the future, so a planner has nothing to query.

UniSim (Yang et al., 2023) learned an interactive simulator from mixed datasets. It accepts instructions such as "open the drawer" and low-level controls. Policies trained purely inside it were deployed in the real world zero-shot.

Wayve's GAIA-4 (2026) closes the loop for driving. If the driving policy brakes, the simulated view slows. A planner can ask what happens if it brakes now.

NVIDIA's Cosmos (2025) is a family of open-weight world foundation models. Teams post-train them on their own data. The platform also ships its video tokenizers and a data curation pipeline.

Watch for three failures. Objects appear or vanish between frames. Actions leak, so the scene moves the same whatever input you give.

Errors compound over a long rollout, because each output becomes the next input.

Take vegetation management on a distribution line. A crew wants to know which trees could reach the conductor in high wind. A video model trained on drone passes could show a span under stronger wind. It is useful only if the wind input really drives the motion.

Test that directly. Hold the scene fixed and vary only the wind. Then compare the predicted sway with a physics-based estimate for that tree and span.

If the sway barely changes, the wind input is decoration. If it changes the wrong way, the model learned the camera and missed the physics. Either result ends the evaluation early.

Measure in this order:

Action sensitivity, the change in output per change in action. - Consistency over time, by tracking objects across the rollout. - Downstream value, whether a planner using the model decides better. - Visual fidelity scores such as FVD, last.

My CIO

Video models are the most visible part of the world-model wave. They also attract the largest claims. A clip that looks real is easy to show in a board meeting. It tells you little about whether the model can support a decision.

Judge them on two things. The first is controllability. Does the scene respond correctly to the action you give it?

The second is consistency. Does it stay coherent after a minute? Many clips that impress at ten seconds fall apart long before that.

Cinematography is the wrong test. For industrial use, a cheaper latent model can win. It only has to answer "what if" correctly. A beautiful model that cannot be steered loses that contest.

The money follows the same line. Pixel generation is heavy on compute. In my course material, a pixel world model planned one robot action in 4 minutes. A feature-space model took 16 seconds.

Some models, such as NVIDIA Cosmos, need NVIDIA GPUs. That shapes where they can run.

The risk is buying a demo. A vendor can show a lovely clip of a storm moving over a service territory. The question is whether the storm moves differently when you change the input. If it does not, you are looking at a film.

The decision is where to spend first. I would recommend to fund a controllability test before any visual polish. It is cheap to run. It settles most of the argument in a week.

Ask the vendor one question. When we change only the action, does the scene change the way reality would?

Quick check

Key terms