-
Mission · Sonu, 10
Latent World Models
Squeeze the world into a small private summary, then predict how the summary changes, not every pixel.
4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea 1 of 4
Imagine you are on the phone with Grandma during your school football match. She cannot see the field. You cannot tell her about every blade of grass. So you tell her the few things that matter.
The score is two to one. Our team has the ball. Maya is running fast toward the goal.
-
Idea 2 of 4
Keep in your head Imagine you are on the phone with Grandma during your school football match.
From those few facts, Grandma can guess what happens next. Maya might shoot. The other team might rush back. She does not need a picture of the whole field to guess.
A latent world model works like your phone call. A camera sees a huge picture made of millions of tiny dots. The computer squeezes that picture into a short list of numbers. The list keeps only what matters, like where the ball is and its speed.
Try this Close the screen. Tell someone at home what this card says, in your own words.
-
Idea 3 of 4
Keep in your head From those few facts, Grandma can guess what happens next.
Then the computer plays a guessing game with its short list. What will the list say one moment from now? And the moment after that?
Guessing a short list is much easier than guessing a whole picture. So the computer can imagine far ahead.
There is one trap. Say you told Grandma "same as before" every single time. Your report would never be wrong. It would also be useless.
-
Idea 4 of 4
Keep in your head Then the computer plays a guessing game with its short list.
A computer can fall into the same trap. So it keeps checking its short list against what the camera really sees.
Try this with a friend at school. Look at a picture in a book for ten seconds. Describe it in only five words.
Your friend draws it from your five words. Which words helped most? What did you leave out that you wish you had kept?
Try this Draw this card as one small picture. Use three lines or fewer.
-
Words to keep
Try this Cover the list. Say each word and what it means, then check.
-
Quick check 1 of 3
-
Quick check 2 of 3
-
Quick check 3 of 3
-
Recap
What you just learned
- Imagine you are on the phone with Grandma during your school football match.
- From those few facts, Grandma can guess what happens next.
- Then the computer plays a guessing game with its short list.
- A computer can fall into the same trap.
-
Mission · Senior AI Architect
Latent World Models
Squeeze the world into a small private summary, then predict how the summary changes, not every pixel.
7 idea cards, 3 questions and a recap. About 7 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea 1 of 7
Pixels are the wrong place to predict. Most of a frame is texture an agent cannot use or predict. A latent world model encodes each observation into a compact state. It learns the dynamics there.
PlaNet (Hafner et al., 2018) introduced the recurrent state-space model, or RSSM. It splits the state in two. A deterministic recurrent part h_t carries memory. A stochastic part z_t carries uncertainty about what the agent cannot see.
-
Idea 2 of 7
Keep in your head Pixels are the wrong place to predict.
Two distributions sit on z_t. The posterior q(z_t | h_t, o_t) sees the current observation. The prior p(z_t | h_t) must predict it blind. Training minimises a prediction loss plus a KL term between the two.
That KL term pulls the prior toward the posterior. At imagination time only the prior runs. Its quality sets the ceiling on planning.
Try this Look away and say this card back in one sentence.
-
Idea 3 of 7
Keep in your head Two distributions sit on zt.
Two failures are common. The KL term can squeeze the latent until it carries no information. Or the prior drifts from the posterior over long horizons.
The fixes are known. Dreamer clips the KL below 3 free nats, as PlaNet did. PlaNet also trained on multi-step predictions, which it called latent overshooting. DreamerV2 (Hafner et al., 2020) moved to discrete categorical latents and added KL balancing.
-
Idea 4 of 7
Keep in your head Two failures are common.
DreamerV3 (Hafner et al., 2023) kept that design. It ran over 150 tasks with one configuration. Nature published it in 2025.
A second branch drops pixel reconstruction entirely. I-JEPA (Assran et al., 2023) predicts the embeddings of hidden image blocks. V-JEPA 2 (2025) does the same on video. Capacity no longer goes to unpredictable texture.
Try this Name one system you have shipped where this would break first.
-
Idea 5 of 7
Keep in your head DreamerV3 (Hafner et al., 2023) kept that design.
The new failure there is collapse. Every input maps to the same embedding and the loss looks perfect. JEPA methods need an explicit guard against it.
Take a feeder model. It reads 16 meter-cluster loads and the outdoor temperature each hour. That is 17 numbers.
-
Idea 6 of 7
Keep in your head The new failure there is collapse.
Air-conditioning load follows indoor heat build-up, a slow average of past temperatures. No single hour shows it.
An encoder squeezes the 17 numbers into a 4-number code. A recurrent memory learns to carry the heat build-up that nobody labelled. A planner can now test a 2 MW transfer between feeders in code space. It never has to redraw 16 load curves.
Try this Find "encoder" on this card. Define it in one line without looking.
-
Idea 7 of 7
Keep in your head Air-conditioning load follows indoor heat build-up, a slow average of past temperatures.
Measure these before you trust the latent:
- Prior-to-posterior KL along the horizon. It shows where imagination stops tracking reality. - Open-loop error at 1, 12 and 48 steps ahead of the last reading. It shows how far the model holds. - A linear probe from latent to feeder load. It shows whether the code tracks what operators watch.
-
Words to keep
Try this Cover the list. Say each word and what it means, then check.
-
Quick check 1 of 3
-
Quick check 2 of 3
-
Quick check 3 of 3
-
Recap
What you just learned
- Pixels are the wrong place to predict.
- Two distributions sit on zt.
- Two failures are common.
- DreamerV3 (Hafner et al., 2023) kept that design.
- The new failure there is collapse.
- Air-conditioning load follows indoor heat build-up, a slow average of past temperatures.
- Measure these before you trust the latent: Prior-to-posterior KL along the horizon.
-
Mission · My CIO
Latent World Models
Squeeze the world into a small private summary, then predict how the summary changes, not every pixel.
4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.
Arrow keys or swipe to move.
-
Idea 1 of 4
Latent models are what make world models cheap enough to use. The model predicts a small hidden summary of the world. It never redraws the full picture. That is why most practical control systems work this way.
The saving is real. In my course material, a robot planner imagined in feature space. It took 16 seconds per action.
-
Idea 2 of 4
Keep in your head Latent models are what make world models cheap enough to use.
A pixel world model took 4 minutes for the same job. That gap decides whether a model can sit inside a live decision loop.
The trade-off is interpretability. The latent is a list of numbers that no person can read. Nobody can open it and find "heat build-up" written on a dial. If the model drives a switching or dispatch decision, someone must show what it tracks.
Try this Name the one decision this card changes for you.
-
Idea 3 of 4
Keep in your head A pixel world model took 4 minutes for the same job.
Two tools do that. A decoder turns the latent back into readings an engineer recognises. A probe tests whether feeder load can be read straight off the latent. Both are cheap compared with the model itself.
The risk without them is quiet. A latent can look fine on average. It can still miss the one variable that matters on a heat-wave afternoon. Nobody will notice until the forecast fails on the worst day.
-
Idea 4 of 4
Keep in your head Two tools do that.
I would recommend to make probes a condition of funding. A team that cannot show what its latent tracks is not ready. It should not let that latent drive a decision yet.
The cost is a few weeks of analyst time. The alternative is a model nobody can audit. Regulators and operators will both ask what it knew and when.
The question to ask the team is this one. What can the hidden state alone tell us about the feeder?
Try this Say what this card would cost you if a vendor got it wrong.
-
Words to keep
Try this Cover the list. Say each word and what it means, then check.
-
Quick check 1 of 3
-
Quick check 2 of 3
-
Quick check 3 of 3
-
Recap
What you just learned
- Latent models are what make world models cheap enough to use.
- A pixel world model took 4 minutes for the same job.
- Two tools do that.
- I would recommend to make probes a condition of funding.