Atlasof World Models
Start

T-09 Province III · The Interior The Navigator’s Bridge 5 min 9 of 21

Model-Based Reinforcement Learning, Planning & Control

How an agent uses a learned world model to decide what to do next, by imagining consequences before acting.

Model-Based Reinforcement Learning, Planning & Control: plan in imagination, act one step, replan Plain words: the pretend board, trying many moves, picking the best plan and making one real move and a mistake on the board that makes some moves look too good. Architect: cross-entropy method rounds that sample, keep elites and refit, a planning horizon H, ensemble disagreement marking the blind spot and model predictive control that executes the first action and replans. CIO: the proposal, the gate that decides what the planner may do alone and the action taken. T-09 · RECEDING HORIZON the pretend board try many moves best plan a mistake on the board check one real move Think several moves ahead. Make one. Look again. model rollouts, horizon H CEM: sample, keep elites mean after 3 rounds ensemble disagrees constraint check MPC: execute a_0, replan Penalise disagreement, or the optimiser finds the model's errors. simulated options options scored proposal too good to be true gate: alone or ask action taken Decide which actions need a person before the planner runs.
A brass compass on a large sheet of paper, a single cobalt line ruled from its point across the page.

In one sentence Imagine the consequences of each option first, then act. The way a chess player thinks a few moves ahead.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

Watch a good chess player before she moves. Her hands stay still for a long time. Inside her head, she is moving pieces that nobody else can see.

If I move my knight here, what will you do? Then what will I do? She thinks three moves ahead in her head. Only then does she touch a real piece.

A beginner plays another way. He picks a move, pushes the piece and sees what happens. Sometimes he loses his queen.

That is a slow way to learn. It costs him pieces too.

Some computers now play the way the good player does. They carry a pretend copy of the game inside them. Before they act, they try many moves on the pretend board. They pick the one that looks best.

Then they make one real move. They look at the real board again. And they start thinking again from there.

There is a catch. What if the pretend board has a mistake in it? Say it thinks a bishop can jump over other pieces.

The computer will find a clever plan that uses that jump. On the pretend board the plan looks perfect. On the real board it fails.

So careful computers check the real board often. They also slow down when their pretend board is unsure about something. A good chess player does the same when a position looks too good.

Try this at school with a friend. Play tic-tac-toe. Before each move, say out loud what your friend will do next.

Keep count of how often you were right. Did thinking ahead help you win?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

A learned model enters control in two ways. It can generate imagined rollouts that train a policy offline, as DreamerV3 does. Or it can be queried at decision time to simulate candidate action sequences.

That second mode is model predictive control, or MPC. Tree search is its close cousin. The first mode moves cost into training. The second spends compute at inference and adapts to states the training data never covered.

MPC is old and well documented. Rawlings, Mayne and Diehl give the theory in their 2017 second edition. Optimise actions over a horizon and execute the first one. Then observe and plan again.

Learning swaps the hand-built model for a neural one. That is where the risks below come from.

PETS (Chua et al., 2018) used an ensemble of probabilistic networks with the cross-entropy method. On half-cheetah it matched model-free results with far fewer samples. The paper reports 8 times fewer than SAC and 125 times fewer than PPO.

TD-MPC2 (Hansen et al., 2023) plans in the latent space of a decoder-free model. One set of hyperparameters covered 104 tasks.

V-JEPA 2-AC (Assran et al., 2025) runs the same loop in video features. It samples 800 candidate moves and keeps the best 10. It refits and repeats for 10 rounds. Then it acts once and plans again.

The central risk is model exploitation. An optimiser searches for actions that score well inside the model. Model errors look like free reward, so it finds them first. A plan can look excellent in imagination and fail on contact with reality.

Ensembles and uncertainty penalties help. So does frequent grounding in real data. MBPO (Janner et al., 2019) limits the damage with short model rollouts. Each rollout branches from a real state.

Take service restoration after a fault on a feeder. The controller proposes switching sequences that move load to a neighbouring feeder. A learned load model scores each sequence.

Suppose its training data never held that neighbour near its thermal limit.

The optimiser will push load there. The model has no reason to object.

An ensemble that disagrees in that region would flag the plan. A hard rule that caps transfers at the engineering limit would block it outright.

Measure on the plans you execute. Random test states hide exploitation, because the planner never chose them. Track these:

  • Predicted against realised outcome for every executed plan. - Ensemble disagreement on the chosen plan, compared with typical states. - How often a hard constraint stopped an action the model liked.

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

This is where a world model stops being research and starts making decisions. The cost of being wrong rises with it.

A planner uses the model to imagine each option before it acts. That saves real trials. In utility operations a real trial can mean a truck roll or a customer outage. It also means the plan is only as good as the model.

The specific risk has a name, model exploitation. A planner hunts for the option that scores best inside the model. Where the model is wrong, it often looks generous. So the planner finds those errors first and calls them good plans.

Two cost lines matter. Training-time imagination costs compute up front and is cheap per decision. Decision-time planning costs compute on every decision. In return it adapts better to situations nobody planned for.

Neither one removes the need for a gate. Somebody must decide which actions a planner may take alone. Somebody must decide which ones need a person. A hard rule at the engineering limit is cheap insurance.

The diligence question changes too. Asking how good the model's imagination is will not catch this failure. Ask what happens when the model is confidently wrong. Ask how that gets detected before it drives a real decision.

Any serious proposal in this area should arrive with an answer. A proposal that assumes it will not happen is not ready. I would recommend to send it back.

The question to ask is this one. When this model is confidently wrong, what catches it before customers feel it?

Quick check

3 questions. Pick one answer each. Get them all right to mark this territory as checked.

1In model predictive control, what does the controller execute after planning?

2What is model exploitation?

3How does MBPO limit the damage from model errors?

Key terms

Further reading

  1. DreamerV3 Danijar Hafner, 2023 Imagination-based training at scale, the offline half of this territory.
  2. Model Predictive Control: Theory, Computation, and Design (2nd edition) James B. Rawlings, David Q. Mayne, Moritz M. Diehl, 2017 The classical MPC literature in one free book. The online-planning half of this territory.
  3. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models (PETS) Kurtland Chua, Roberto Calandra, Rowan McAllister, Sergey Levine, 2018 Ensembles plus the cross-entropy method. 8 and 125 times fewer samples than SAC and PPO on half-cheetah.
  4. When to Trust Your Model: Model-Based Policy Optimization (MBPO) Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine, 2019 Short model rollouts branched from real data, a direct answer to model exploitation.
  5. TD-MPC2: Scalable, Robust World Models for Continuous Control Nicklas Hansen, Hao Su, Xiaolong Wang, 2023 Planning in a decoder-free latent across 104 tasks with one set of hyperparameters.
  6. V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning Mido Assran et al., 2025 V-JEPA 2-AC plans robot moves with the cross-entropy method inside MPC, in feature space.

Search the atlas