T-16 Province V · The Frontier The Training Yard 5 min 16 of 21 New
World Models for Robots I: Simulation, Control & Sim-to-Real
Training robots in simulated or learned worlds, and the craft of making those skills survive contact with reality.
In one sentence Practise in a pretend world full of variation, so the real world is just one more variation.
Pilots practise in flight simulators before they fly a real plane. A crash in a simulator costs nothing. Robots practise the same way, inside a pretend world on a computer.
Picture a small robot dog learning to walk. In the pretend world it takes a million wobbly steps. It falls over thousands of times and nobody minds.
Then we put it on the real kitchen floor. It slips.
Why did it slip? The pretend world was never quite like the kitchen. The real floor is slipperier and the real motor is a bit weaker.
Even the light looks different to its camera.
Robot builders have a clever trick for this. They make the pretend world messy on purpose. Every time the dog practises, the floor changes.
Sometimes it is icy. Sometimes it is sticky like honey.
The dog's legs get heavier, then lighter. The colours of the room change too. After a thousand strange floors, one more does not scare it. The kitchen floor is just another strange floor.
There is a second trick. Some robots skip most of the pretend world and learn from the real one. A project called DayDreamer did this with a real robot dog.
It learned to roll off its back and stand up. Then it learned to walk, all in about one hour.
How so fast? It also practised inside its own head. It built a small picture of the world from what it felt. Then it rehearsed many extra steps in that picture.
Try this at home. Walk across your room in socks. Walk back in shoes, then barefoot. Notice how your feet grip in a different way each time.
You did not need a lesson for each floor. That is what the messy pretend world teaches a robot.
The sim-to-real gap has two parts. Dynamics mismatch means the simulated physics differs from the real plant. Perception mismatch means simulated sensors see a cleaner world than real cameras do.
Each part has its own fix and its own way to fail.
System identification fits simulator parameters to real measurements. Domain randomization takes the other road. It trains across wide parameter distributions so the real system falls inside them.
One detector learned only from simulated images with random textures (Tobin et al., 2017). It located real objects to within 1.5 cm. Another team randomized the physics instead (Peng et al., 2017). Their pushing policy trained only in simulation and held its performance on a real arm.
Learned world models offer a third path. You fit dynamics to real data and plan or train inside that model. DayDreamer (Wu et al., 2022) ran Dreamer on four physical robots. A quadruped learned to stand and walk from scratch in about one hour.
It adapted to pushes within 10 minutes. DINO-WM (Zhou et al., 2024) plans on frozen DINOv2 patch features. It solved six environments zero-shot, without expert demonstrations.
Control then takes one of two shapes. Model predictive control re-plans every step with the model. A distilled policy learns from imagined rollouts and runs fast at the edge. Real-world fine-tuning with a small amount of data closes the remaining gap.
The trade-offs are real. Randomize too widely and the policy turns timid. Randomize too narrowly and reality sits outside the training range. System identification overfits to the day you measured.
A learned model drifts over long rollouts. A planner will exploit any error it finds. The failure I see most often is time.
Simulators deliver every observation at once. Real sensor data arrives late, out of order or never.
Take a restoration agent trained on a simulated distribution feeder. In the simulator every meter reports the moment a fault hits. In the field, last-gasp messages from meters arrive late or never. SCADA polls on a cycle and crew updates come by phone.
I would recommend randomizing exactly those delays and dropouts. Fit their ranges to last year's outage logs. That is system identification on the timing.
Then replay a held-out storm. Compare the agent's switching plans against what operators did. The agent proposes and the operator switches.
Measure four numbers before you trust a transfer:
- real data needed to reach the target score
- the drop from simulator to unseen real conditions
- interventions per hour during on-site learning
- how often the agent acts on stale state
Safety during on-site learning is an engineering requirement. Set action limits and keep a human override. Explore conservatively.
Real robot hours are slow and expensive. Simulation and world models are what make robot learning affordable. A robot can fall a million times in software for the price of electricity.
The risk sits at the handover. A system that looks excellent in simulation can fail on its first real site. The floor, the weather and the sensor delays differ from the model.
When that happens you pay in equipment and schedule. Sometimes you pay in safety.
The same logic applies well beyond robots. Any agent trained on synthetic data faces this gap. A restoration agent trained on a simulated feeder meets real meters that report late.
A billing agent trained on clean synthetic accounts meets real corrections. It also meets estimated reads.
So the money question is where to spend. One option is to build a better simulator. Another is to collect more real data.
The third is to fund careful on-site fine-tuning under supervision. Strong teams do all three in proportion. They can tell you which one buys the most improvement right now.
Ask any vendor or internal team for three answers:
- real data needed to reach target performance
- the performance drop on unseen conditions
- the safety envelope during learning on site
A team that cannot answer the second question has not measured the gap. It has only measured the simulator.
I would treat simulator results as a hypothesis. The first real site is the experiment. Budget for it explicitly, with a rollback plan.
Name a person who can stop the system. Then ask one question. How far does performance drop from our simulator to our first real site?
Quick check
Key terms
Further reading
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World Tobin, Fong, Ray, Schneider, Zaremba, Abbeel, 2017 The randomization recipe. A detector trained on random textures found real objects to within 1.5 cm.
- Sim-to-Real Transfer of Robotic Control with Dynamics Randomization Peng, Andrychowicz, Zaremba, Abbeel, 2017 Randomizes physics instead of pixels and transfers a pushing policy to a real arm.
- DayDreamer: World Models for Physical Robot Learning Wu, Escontrela, Hafner, Goldberg, Abbeel, 2022 A world model learned on real robots. A quadruped walks from scratch in about one hour.
- DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning Zhou, Pan, LeCun, Pinto, 2024 Planning on frozen DINOv2 features. ICML 2025 version at proceedings.mlr.press/v267/zhou25t.html.