Atlasof World Models
Start

T-09 · Model-Based Reinforcement Learning, Planning & Control

  1. Mission · Sonu, 10

    Model-Based Reinforcement Learning, Planning & Control

    Imagine the consequences of each option first, then act. The way a chess player thinks a few moves ahead.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Watch a good chess player before she moves. Her hands stay still for a long time. Inside her head, she is moving pieces that nobody else can see.

    If I move my knight here, what will you do? Then what will I do? She thinks three moves ahead in her head. Only then does she touch a real piece.

  3. Idea 2 of 4

    Keep in your head Watch a good chess player before she moves.

    A beginner plays another way. He picks a move, pushes the piece and sees what happens. Sometimes he loses his queen.

    That is a slow way to learn. It costs him pieces too.

    Some computers now play the way the good player does. They carry a pretend copy of the game inside them. Before they act, they try many moves on the pretend board. They pick the one that looks best.

    Try this Close the screen. Tell someone at home what this card says, in your own words.

  4. Idea 3 of 4

    Keep in your head A beginner plays another way.

    Then they make one real move. They look at the real board again. And they start thinking again from there.

    There is a catch. What if the pretend board has a mistake in it? Say it thinks a bishop can jump over other pieces.

    The computer will find a clever plan that uses that jump. On the pretend board the plan looks perfect. On the real board it fails.

  5. Idea 4 of 4

    Keep in your head Then they make one real move.

    So careful computers check the real board often. They also slow down when their pretend board is unsure about something. A good chess player does the same when a position looks too good.

    Try this at school with a friend. Play tic-tac-toe. Before each move, say out loud what your friend will do next.

    Keep count of how often you were right. Did thinking ahead help you win?

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1In model predictive control, what does the controller execute after planning?

  8. Quick check 2 of 3

    2What is model exploitation?

  9. Quick check 3 of 3

    3How does MBPO limit the damage from model errors?

  10. Recap

    What you just learned

    1. Watch a good chess player before she moves.
    2. A beginner plays another way.
    3. Then they make one real move.
    4. So careful computers check the real board often.

  1. Mission · Senior AI Architect

    Model-Based Reinforcement Learning, Planning & Control

    Imagine the consequences of each option first, then act. The way a chess player thinks a few moves ahead.

    7 idea cards, 3 questions and a recap. About 7 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 7

    A learned model enters control in two ways. It can generate imagined rollouts that train a policy offline, as DreamerV3 does. Or it can be queried at decision time to simulate candidate action sequences.

    That second mode is model predictive control, or MPC. Tree search is its close cousin. The first mode moves cost into training. The second spends compute at inference and adapts to states the training data never covered.

  3. Idea 2 of 7

    Keep in your head A learned model enters control in two ways.

    MPC is old and well documented. Rawlings, Mayne and Diehl give the theory in their 2017 second edition. Optimise actions over a horizon and execute the first one. Then observe and plan again.

    Learning swaps the hand-built model for a neural one. That is where the risks below come from.

    Try this Look away and say this card back in one sentence.

  4. Idea 3 of 7

    Keep in your head MPC is old and well documented.

    PETS (Chua et al., 2018) used an ensemble of probabilistic networks with the cross-entropy method. On half-cheetah it matched model-free results with far fewer samples. The paper reports 8 times fewer than SAC and 125 times fewer than PPO.

    TD-MPC2 (Hansen et al., 2023) plans in the latent space of a decoder-free model. One set of hyperparameters covered 104 tasks.

  5. Idea 4 of 7

    Keep in your head PETS (Chua et al., 2018) used an ensemble of probabilistic networks with the cross-entropy method.

    V-JEPA 2-AC (Assran et al., 2025) runs the same loop in video features. It samples 800 candidate moves and keeps the best 10. It refits and repeats for 10 rounds. Then it acts once and plans again.

    The central risk is model exploitation. An optimiser searches for actions that score well inside the model. Model errors look like free reward, so it finds them first. A plan can look excellent in imagination and fail on contact with reality.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 7

    Keep in your head V-JEPA 2-AC (Assran et al., 2025) runs the same loop in video features.

    Ensembles and uncertainty penalties help. So does frequent grounding in real data. MBPO (Janner et al., 2019) limits the damage with short model rollouts. Each rollout branches from a real state.

    Take service restoration after a fault on a feeder. The controller proposes switching sequences that move load to a neighbouring feeder. A learned load model scores each sequence.

  7. Idea 6 of 7

    Keep in your head Ensembles and uncertainty penalties help.

    Suppose its training data never held that neighbour near its thermal limit.

    The optimiser will push load there. The model has no reason to object.

    An ensemble that disagrees in that region would flag the plan. A hard rule that caps transfers at the engineering limit would block it outright.

    Try this Write down the one number on this card you would put on a dashboard.

  8. Idea 7 of 7

    Keep in your head Suppose its training data never held that neighbour near its thermal limit.

    Measure on the plans you execute. Random test states hide exploitation, because the planner never chose them. Track these:

    • Predicted against realised outcome for every executed plan. - Ensemble disagreement on the chosen plan, compared with typical states. - How often a hard constraint stopped an action the model liked.
  9. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  10. Quick check 1 of 3

    1In model predictive control, what does the controller execute after planning?

  11. Quick check 2 of 3

    2What is model exploitation?

  12. Quick check 3 of 3

    3How does MBPO limit the damage from model errors?

  13. Recap

    What you just learned

    1. A learned model enters control in two ways.
    2. MPC is old and well documented.
    3. PETS (Chua et al., 2018) used an ensemble of probabilistic networks with the cross-entropy method.
    4. V-JEPA 2-AC (Assran et al., 2025) runs the same loop in video features.
    5. Ensembles and uncertainty penalties help.
    6. Suppose its training data never held that neighbour near its thermal limit.
    7. Measure on the plans you execute.

  1. Mission · My CIO

    Model-Based Reinforcement Learning, Planning & Control

    Imagine the consequences of each option first, then act. The way a chess player thinks a few moves ahead.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    This is where a world model stops being research and starts making decisions. The cost of being wrong rises with it.

    A planner uses the model to imagine each option before it acts. That saves real trials. In utility operations a real trial can mean a truck roll or a customer outage. It also means the plan is only as good as the model.

  3. Idea 2 of 4

    Keep in your head This is where a world model stops being research and starts making decisions.

    The specific risk has a name, model exploitation. A planner hunts for the option that scores best inside the model. Where the model is wrong, it often looks generous. So the planner finds those errors first and calls them good plans.

    Two cost lines matter. Training-time imagination costs compute up front and is cheap per decision. Decision-time planning costs compute on every decision. In return it adapts better to situations nobody planned for.

    Try this Find "planning" on this card. Define it in one line without looking.

  4. Idea 3 of 4

    Keep in your head The specific risk has a name, model exploitation.

    Neither one removes the need for a gate. Somebody must decide which actions a planner may take alone. Somebody must decide which ones need a person. A hard rule at the engineering limit is cheap insurance.

    The diligence question changes too. Asking how good the model's imagination is will not catch this failure. Ask what happens when the model is confidently wrong. Ask how that gets detected before it drives a real decision.

  5. Idea 4 of 4

    Keep in your head Neither one removes the need for a gate.

    Any serious proposal in this area should arrive with an answer. A proposal that assumes it will not happen is not ready. I would recommend to send it back.

    The question to ask is this one. When this model is confidently wrong, what catches it before customers feel it?

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1In model predictive control, what does the controller execute after planning?

  8. Quick check 2 of 3

    2What is model exploitation?

  9. Quick check 3 of 3

    3How does MBPO limit the damage from model errors?

  10. Recap

    What you just learned

    1. This is where a world model stops being research and starts making decisions.
    2. The specific risk has a name, model exploitation.
    3. Neither one removes the need for a gate.
    4. Any serious proposal in this area should arrive with an answer.

Search the atlas