Atlasof World Models
Start

T-08 Province III · The Interior The Hidden Valley 5 min 8 of 21 New

Latent World Models

Why the best world models predict in a small hidden space rather than in raw pixels, and how they keep that space honest.

Latent World Models: predicting in a short latent instead of pixels Plain words: the whole field is squeezed into a short report, the report is checked whenever the camera peeks and it gets less sure the further it guesses alone. Architect: recurrent state h_t and stochastic state z_t, posterior q(z_t | h_t, o_t) when a frame arrives, prior p(z_t | h_t) in imagination, a KL term pulling the prior to the posterior, drift over the horizon. CIO: cheap to run, cannot be read directly, probe here and quiet drift that nobody sees without a probe. T-08 · LATENT ROLLOUT t0t1t2t3t4t5t6 the whole field the camera peeks guessing ahead, eyes shut the short report less sure the further it guesses Squeeze the field into a short report. Guess the report, not the picture. ot 64 x 64 x 3 h z posterior q(z | h, o) KL prior p(z | h) imagination, no frames RSSM: h + z prior drifts from posterior Only the prior runs at imagination time. Its quality caps planning. camera frame real data cheap to run 16 s per action, not 4 min cannot be read directly probe here quiet drift, nobody sees it Fund the probe that shows what the latent tracks.
A block of clear glass on a table with one cobalt sphere suspended inside it, a lake beyond the window.

In one sentence Squeeze the world into a small private summary, then predict how the summary changes, not every pixel.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

Imagine you are on the phone with Grandma during your school football match. She cannot see the field. You cannot tell her about every blade of grass. So you tell her the few things that matter.

The score is two to one. Our team has the ball. Maya is running fast toward the goal.

From those few facts, Grandma can guess what happens next. Maya might shoot. The other team might rush back. She does not need a picture of the whole field to guess.

A latent world model works like your phone call. A camera sees a huge picture made of millions of tiny dots. The computer squeezes that picture into a short list of numbers. The list keeps only what matters, like where the ball is and its speed.

Then the computer plays a guessing game with its short list. What will the list say one moment from now? And the moment after that?

Guessing a short list is much easier than guessing a whole picture. So the computer can imagine far ahead.

There is one trap. Say you told Grandma "same as before" every single time. Your report would never be wrong. It would also be useless.

A computer can fall into the same trap. So it keeps checking its short list against what the camera really sees.

Try this with a friend at school. Look at a picture in a book for ten seconds. Describe it in only five words.

Your friend draws it from your five words. Which words helped most? What did you leave out that you wish you had kept?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

Pixels are the wrong place to predict. Most of a frame is texture an agent cannot use or predict. A latent world model encodes each observation into a compact state. It learns the dynamics there.

PlaNet (Hafner et al., 2018) introduced the recurrent state-space model, or RSSM. It splits the state in two. A deterministic recurrent part h_t carries memory. A stochastic part z_t carries uncertainty about what the agent cannot see.

Two distributions sit on z_t. The posterior q(z_t | h_t, o_t) sees the current observation. The prior p(z_t | h_t) must predict it blind. Training minimises a prediction loss plus a KL term between the two.

That KL term pulls the prior toward the posterior. At imagination time only the prior runs. Its quality sets the ceiling on planning.

Two failures are common. The KL term can squeeze the latent until it carries no information. Or the prior drifts from the posterior over long horizons.

The fixes are known. Dreamer clips the KL below 3 free nats, as PlaNet did. PlaNet also trained on multi-step predictions, which it called latent overshooting. DreamerV2 (Hafner et al., 2020) moved to discrete categorical latents and added KL balancing.

DreamerV3 (Hafner et al., 2023) kept that design. It ran over 150 tasks with one configuration. Nature published it in 2025.

A second branch drops pixel reconstruction entirely. I-JEPA (Assran et al., 2023) predicts the embeddings of hidden image blocks. V-JEPA 2 (2025) does the same on video. Capacity no longer goes to unpredictable texture.

The new failure there is collapse. Every input maps to the same embedding and the loss looks perfect. JEPA methods need an explicit guard against it.

Take a feeder model. It reads 16 meter-cluster loads and the outdoor temperature each hour. That is 17 numbers.

Air-conditioning load follows indoor heat build-up, a slow average of past temperatures. No single hour shows it.

An encoder squeezes the 17 numbers into a 4-number code. A recurrent memory learns to carry the heat build-up that nobody labelled. A planner can now test a 2 MW transfer between feeders in code space. It never has to redraw 16 load curves.

Measure these before you trust the latent:

  • Prior-to-posterior KL along the horizon. It shows where imagination stops tracking reality. - Open-loop error at 1, 12 and 48 steps ahead of the last reading. It shows how far the model holds. - A linear probe from latent to feeder load. It shows whether the code tracks what operators watch.

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

Latent models are what make world models cheap enough to use. The model predicts a small hidden summary of the world. It never redraws the full picture. That is why most practical control systems work this way.

The saving is real. In my course material, a robot planner imagined in feature space. It took 16 seconds per action.

A pixel world model took 4 minutes for the same job. That gap decides whether a model can sit inside a live decision loop.

The trade-off is interpretability. The latent is a list of numbers that no person can read. Nobody can open it and find "heat build-up" written on a dial. If the model drives a switching or dispatch decision, someone must show what it tracks.

Two tools do that. A decoder turns the latent back into readings an engineer recognises. A probe tests whether feeder load can be read straight off the latent. Both are cheap compared with the model itself.

The risk without them is quiet. A latent can look fine on average. It can still miss the one variable that matters on a heat-wave afternoon. Nobody will notice until the forecast fails on the worst day.

I would recommend to make probes a condition of funding. A team that cannot show what its latent tracks is not ready. It should not let that latent drive a decision yet.

The cost is a few weeks of analyst time. The alternative is a model nobody can audit. Regulators and operators will both ask what it knew and when.

The question to ask the team is this one. What can the hidden state alone tell us about the feeder?

Quick check

3 questions. Pick one answer each. Get them all right to mark this territory as checked.

1Why does a latent world model predict a short code and skip the full picture?

2In an RSSM, which part runs alone when the agent imagines the future?

3What goes wrong when the KL term squeezes the latent too hard?

Key terms

Further reading

  1. Learning Latent Dynamics for Planning from Pixels (PlaNet) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, James Davidson, 2018 Introduced the RSSM with its deterministic and stochastic parts, plus latent overshooting.
  2. Mastering Atari with Discrete World Models (DreamerV2) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, Jimmy Ba, 2020 Discrete categorical latents and KL balancing. First agent at human level on 55 Atari tasks by learning inside a separately trained world model.
  3. Mastering Diverse Domains through World Models (DreamerV3) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap, 2023 One latent model and one configuration across over 150 tasks. Published in Nature in 2025.
  4. Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA) Mahmoud Assran et al., 2023 Predicts embeddings of hidden blocks and never reconstructs pixels.
  5. Deep Dive into Yann LeCun's JEPA Rohit Bandaru, 2024 A readable account of predicting latents without reconstruction.

Search the atlas