T-03 Province I · The Founding Coast The Coastline 5 min 3 of 21
Environments, Simulators & Rollouts
The interface between a learned model and the environment it is trying to predict, and what a "rollout" actually is.
In one sentence Pressing play on imagination is easy. Keeping the daydream honest over many steps is the hard part.
Put a dot on one side of a sheet of paper. Put a star on the other side. Now close your eyes and draw a straight line from the dot to the star.
For the first bit, your hand feels sure. The line looks fine. But your hand wobbles a tiny bit at the start. You cannot see it.
That tiny wobble keeps going. Each bit of line starts from where the last bit ended. By the end, you might miss the star by a whole hand.
Machines have the same problem when they daydream. A world model is a picture of how things work. A rollout is pressing play on that picture. The machine guesses one step, then the next step from its own guess.
Roll it out ten steps and you get a short daydream. Roll it out a thousand steps and you get a whole plan. Small mistakes early on grow into big, strange mistakes later.
A ball that should bounce once might start sinking through the floor. Nobody told the machine it was wrong. So it kept going.
Now draw the line again. This time, peek every two seconds. Each peek lets you fix your hand.
Scientists call that closed-loop. Drawing with your eyes shut the whole way is open-loop.
Machines that peek at the real world now and then stay honest longer. Keeping a long daydream honest is one of the hardest problems in this field.
Try this at home or at school. Draw the dot and the star about as far apart as your hand is long. Draw once with eyes shut the whole way. Then draw once with a peek every two seconds.
Use a ruler to measure how far each line ends from the star. Which one won?
An environment interface fixes how a model and the world exchange information. An observation comes in and an action goes out. A reward and the next observation come back.
OpenAI Gym (Brockman et al., 2016) made this loop the default. Gymnasium (Towers et al., 2024) now maintains the standard API.
A rollout is a trajectory produced by running that loop many times. In an open-loop rollout the model's own predictions feed its next input. In a closed-loop rollout ground truth is injected at each step to correct drift.
Open-loop rollouts are what make imagination training possible. A policy can practise entirely inside the model. They are also where compounding error lives. Any per-step bias multiplies across the horizon.
Most of the craft here is horizon management. Teacher forcing trains on true inputs, which hides the problem. Scheduled sampling (Bengio et al., 2015) shifts training toward the model's own outputs over time. Rollout-length curricula wean the model off ground truth without letting it destabilize.
Two published answers are worth knowing. MBPO (Janner et al., 2019) keeps model rollouts short and branches them from real states. DreamerV3 (Hafner et al., Nature 2025) trains an actor and a critic inside imagined rollouts. One configuration covered more than 150 tasks.
DreamerV3 was also the first algorithm to collect diamonds in Minecraft from scratch. It used no human data and no curriculum. Its learned value at the end of a short dream reaches far past that horizon.
Here is the feeder version, from a toy lab of mine. Two feeders each report 8 meter-cluster loads every hour, plus outdoor temperature. The action moves 0 or 2 MW between the feeders.
A hidden heat build-up drives air-conditioning load. No single hour shows it.
Each model saw 12 real hours. Then it got only the actions for the next 48 hours and dreamed. The open-loop GRU had the best one-step error. Its 48-hour error was the worst, 2.19 in normalized units.
Training it to predict the change cut that to 0.15. Adding closed-loop practice cut it to 0.06. That is a single seed, so the exact numbers deserve a loose grip. The shape of the result is the lesson.
What should you measure? Plot error against horizon, out to the length the planner uses. Find the horizon where the plan stops being useful. Then test the planner's choices as well as the frames.
Watch for model exploitation as well. A planner will search for actions where the model is wrong in its favor. Bound the actions to the range the model has seen.
This is the territory to question hardest before trusting a vendor demo. A demo usually shows a short rollout. A model can look flawless at five steps and fall apart at fifty.
If the demo never shows fifty steps, assume the drift is there. That assumption is cheap. Missing real drift is expensive.
Ask one specific thing. How does error grow with horizon length? At what horizon does the model stop being useful for our decisions? That single number says more about production readiness than any demo reel.
The horizon question maps to utility work directly. Day-ahead load planning needs about 24 hourly steps. A storm restoration plan may need several days. A model tuned and tested on one-hour steps tells you little about either.
The commercial upside of getting this right is real. A simulator that stays honest over long horizons changes what a team can say. The team stops saying they modeled it. They start saying they can plan with it.
That is the gap between an interesting demo and a system worth staking decisions on.
There is a cost trade inside the answer. Longer honest rollouts need more training, more checks and often more compute. Shorter rollouts anchored to real data are cheaper and safer. Many teams are better served by the shorter path at first.
I would recommend to make the error curve part of every vendor review. Ask for it on our own data and at our own horizon. Ask for it again after the first season in service.
The question I would ask the vendor is direct. Show me the error curve out to our planning horizon. Where does it cross the line where we stop trusting it?
Quick check
Key terms
Further reading
- OpenAI Gym Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, Wojciech Zaremba, 2016 The observation, action and reward loop the field still uses.
- Gymnasium: A Standard Interface for Reinforcement Learning Environments Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis and others, 2024 The maintained successor to Gym and the API most environments now target.
- Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks Samy Bengio, Oriol Vinyals, Navdeep Jaitly, Noam Shazeer, 2015 The curriculum that moves training from true inputs to the model's own outputs.
- When to Trust Your Model: Model-Based Policy Optimization Michael Janner, Justin Fu, Marvin Zhang, Sergey Levine, 2019 Short model rollouts branched from real data, with the analysis behind that choice.
- Mastering diverse control tasks through world models (DreamerV3) Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap, 2025 Training inside imagined rollouts at scale. First to collect Minecraft diamonds from scratch.