T-07 Province II · The Ateliers The Fresco Rooms 5 min 7 of 21 New
Diffusion & Flow Matching
The generative recipe behind most modern image and video world models: learn to undo noise, one small step at a time.
In one sentence Teach a model to clean up a slightly noisy picture, then let it clean up pure static into something new.
Draw a bright chalk sun on the driveway. Now picture a light rain. Each drop smudges the sun a tiny bit.
After a few drops you can still see the sun. After many drops it turns blurry. In the end it is only a grey smear. Nobody could tell it was ever a sun.
A diffusion model is a computer that learns this rain backward. Scientists show it a smudged picture. Then they ask one small question. What did this look like one drop ago, a little less smudged?
The computer practises that one small step millions of times. It gets very good at un-smudging just a little.
Then comes the surprise. Hand it a pure grey smear that was never a drawing. Ask it to un-smudge again and again.
Step by step a brand new picture appears. Nobody drew it before.
That is how many computer pictures and videos get made today. It takes many small steps, so it can be slow.
Flow matching is a close cousin. Think of the grey smear as a cloud of chalk dust. Flow matching learns which way to push every speck.
The whole cloud then slides smoothly into a drawing. Straighter pushes need fewer steps, so it can be faster.
Some world models use this to imagine the next moment of a game. They un-smudge the future one frame at a time.
Try this at home with a grown-up. Draw a simple sun with a pencil. Take a photo.
Rub it gently with a tissue and take another photo. Do this five times, until it is only a smudge.
Now look at the photos backward, from smudge to sun. That is what the computer learned to do. Could you guess the sun from photo four alone?
Diffusion starts from a fixed forward process. Gaussian noise is added to data over many steps until only noise remains. A network learns to reverse it one step at a time.
Sohl-Dickstein et al. (2015) introduced the idea. Ho et al. (2020) made it practical with DDPM.
The network predicts the noise at each level, which is equivalent to learning the score. DDPM reached an FID of 3.17 on unconditional CIFAR-10.
Latent diffusion (Rombach et al., CVPR 2022) moved the process into a compressed autoencoder latent. That cut compute enough to make high-resolution image generation affordable. Most video work followed the same path.
Flow matching (Lipman et al., 2022) reframes generation as learning a velocity field. The field transports a simple distribution to the data distribution along chosen paths. Training is plain regression on target velocities. Sampling integrates an ODE.
Their optimal transport paths run straight between noise and data. In their tests these trained and sampled faster than diffusion paths. Rectified flow (Liu et al., 2022) straightens paths further. On some image tasks a single Euler step gave good results.
Straighter paths need fewer integration steps. That matters for any world model that must roll out many frames.
For world modeling the extension is conditioning. A model conditions on past frames, on actions or on text. Classifier-free guidance (Ho and Salimans, 2022) mixes conditional and unconditional scores. Higher guidance trades diversity for adherence to the condition.
Two results show what diffusion world models can do. DIAMOND (Alonso et al., NeurIPS 2024) trained agents entirely inside a diffusion world model. It reached a mean human-normalized score of 1.46 on Atari 100k. GameNGen (Valevski et al., 2024) simulated DOOM at 20 frames per second on one TPU.
In the GameNGen study, raters barely beat chance at telling real clips from simulated ones. That says the frames look right. It says less about whether the dynamics are right.
The bottleneck is sampling cost per frame. Step distillation and few-step samplers are an active line of work for that reason.
Here is the feeder case. My feeder lab used a deterministic model. When the future forks, a heat wave breaking or holding, it predicts the average day. That day may never happen.
A diffusion model conditioned on a temperature forecast samples days instead. A planner then sees both branches and the spread between them.
Three failure modes deserve tests. A frame can look perfect and ignore the action. Long autoregressive rollouts can drift. High guidance can collapse diversity and hide the risky branch.
I would recommend to measure action consistency first. Change the action and check that the output changes the right way. Then log steps per frame, latency and coverage of real outcomes.
Diffusion and flow models are the reason generated video now looks convincing. That visual quality is real. It is a different thing from an accurate simulation.
A model can produce a beautiful frame that breaks physics. It can also ignore the action it was asked to show. In a planning tool, that second failure is the dangerous one. The picture looks right and the decision it supports is wrong.
The research results are strong. One team trained game-playing agents entirely inside a diffusion world model. Another ran a playable game simulation at 20 frames per second on a single chip. Human raters struggled to tell its clips from the real game.
Those results show the frames can fool people. They do not prove the model obeys the rules of the system it imitates.
The money sits in sampling. Each frame or sample can take many steps. Steps mean compute. Compute means cost and latency on every call.
Newer flow methods cut the step count, sometimes sharply.
For a utility the fit is clearest in offline work. Scenario studies for storms, heat waves or load growth can afford slow, careful sampling. A sampler that draws many plausible futures shows planners the spread. A single average forecast cannot.
Real-time control is a harder fit today. A switching decision cannot wait on a slow sampler.
When evaluating a vendor built on this stack, ask two things. How many sampling steps does each frame or scenario need? That sets cost and latency. Are outputs checked against action consistency, beyond how they look?
I would recommend to require an action-consistency test in every pilot. It is cheap to run and it catches the failure that matters most.
The question I would ask the vendor is this. If we change the action, does the outcome change the way physics says it should?
Quick check
Key terms
Further reading
- Denoising Diffusion Probabilistic Models Jonathan Ho, Ajay Jain, Pieter Abbeel, 2020 The paper that made diffusion practical. FID 3.17 on CIFAR-10.
- Flow Matching for Generative Modeling Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, Matt Le, 2022 The velocity-field formulation, with optimal transport paths that sample faster.
- High-Resolution Image Synthesis with Latent Diffusion Models Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer, 2022 Diffusion in a compressed latent space, the step that made high resolution affordable.
- Classifier-Free Diffusion Guidance Jonathan Ho, Tim Salimans, 2022 The guidance knob that trades diversity for adherence to the condition.
- Diffusion for World Modeling: Visual Details Matter in Atari Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, François Fleuret, 2024 DIAMOND. Agents trained entirely inside a diffusion world model, 1.46 mean HNS on Atari 100k.
- Diffusion Models Are Real-Time Game Engines Dani Valevski, Yaniv Leviathan, Moab Arar, Shlomi Fruchter, 2024 GameNGen. DOOM simulated at 20 frames per second on one TPU.