Atlasof World Models
Start

T-14 Province IV · The Moving Image The Clockwork Map 5 min 14 of 21 New

Spatial World Models II: 4D Dynamics & Interaction

Adding time to 3D: scenes that move, objects that are tracked, and spaces you can act inside.

Spatial World Models II: 4D Dynamics & Interaction: a scene that moves and answers back Plain words: the dog is still there behind the bench and what happens if you push the gate. Architect: a canonical static space, a deformation over time for what moves, a track ID kept through occlusion and an occupancy forecast that responds to an intervention. CIO: a model that changes over time and responds to action, against a static scan that cannot answer a question about the future. T-14 · TIME, OCCLUSION, ACTION #7 t the dog behind the bench: still there push it Things keep moving when hidden. Push and the future changes. track ID kept through occlusion canonical space + deformation intervene occupancy forecast Static scene once, motion as deformation over time, forecasts that react. changes over time a static scan stops here responds to action A predictive twin can change decisions. A static one cannot.

A round pale column on a bare floor at sunrise, a trail of small marks and one cobalt dot at its base.

In one sentence A 3D map is a photo of a place. A 4D model is a film of it that you can step into.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

A map of a park shows you the paths and the pond. It shows where every tree stands. It does not show the dog running across the grass. It does not show the swings moving.

For that you need time.

A 4D world model is a 3D place that changes over time. People walk. Gates open. A ball rolls under a bench.

Here is the clever part. When the ball is under the bench, you cannot see it. The model still knows it is there.

Grown-ups call this object permanence. You learned it so long ago that you never notice you use it. Peekaboo is a game about it.

The model also follows each thing as it moves. The dog at the gate is the same dog at the pond a moment later. If two dogs cross paths, the model should not mix them up.

That is harder than it sounds. Two brown dogs look alike from far away. The model has to remember which way each one was running.

The hardest part is pushing. Imagine you could reach into the model and push a swing. It should swing.

Push a bench and it should slide or stay put. That depends on how heavy it is. A model that knows those things is more than a pretty picture. It is a world you can think with.

Try this at the playground. Pick one person or one dog. Follow them with your eyes for a minute.

When they go behind something, guess where they will come out. Were you right? What helped you guess?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

Dynamic scenes add a time axis to the 3D methods of the previous territory. D-NeRF (Pumarola et al., 2020) keeps a canonical scene. It learns a time-conditioned deformation from that scene into each moment. It renders new views of moving objects from a single moving camera.

Gaussian methods move the primitives themselves. Dynamic 3D Gaussians (Luiten et al., 2023) let each Gaussian move and rotate over time. Its colour, opacity and size stay fixed.

Local rigidity priors keep neighbours moving together. Dense 6-DOF tracking falls out of the reconstruction with no flow input.

4D Gaussian Splatting (Wu et al., 2023) predicts deformations with a small network. It renders at 82 frames per second at 800 by 800 on an RTX 3090.

Three representations do most of the work:

  • Scene flow estimates 3D motion per point. - Long-term tracking keeps identities through occlusion. - Dynamic occupancy forecasts which space will be filled.

Dynamic occupancy is the workhorse in autonomous driving. OccWorld (Zheng et al., 2023) tokenizes 3D occupancy. A GPT-style transformer then predicts future scenes. It predicts the vehicle's own trajectory alongside them.

Occupancy has practical advantages. It is cheap to get from sparse LiDAR. It works with cameras too. It describes shape at a finer grain than bounding boxes.

Interaction is the frontier. Predicting how a scene responds to pushing, opening or grasping needs more than appearance.

The representation must know where each object ends and how rigid it is. It must also model contact. This is where spatial models meet learned physics and robot learning.

Take storm response. Drone and truck cameras capture a damaged feeder over two days. A 4D model holds the scene as it changes.

A leaning pole tilts further. A crew clears a fallen tree. A section of line goes back up.

Planners can scrub through time and see what changed between passes. The tracking problem is real. The same pole must keep one identity from pass to pass. The angle and the light change every time.

A wrong identity merge is costly. Two poles become one in the record. A crew then drives to the wrong span.

Interaction matters here too. A planner may ask what happens if a crew drops a tree to one side. Today that answer comes from people, with the model as the shared picture. Learned dynamics for that question are still research.

Measure these three:

  • Identity accuracy through occlusion and across capture passes. - Occupancy forecast error at fixed horizons. - Response to an intervention, compared with what really happened.

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

4D models are the bridge between two kinds of digital twin. The first kind looks right. The second kind predicts.

A 3D scan shows where things are. A 4D model shows how they move.

Some of this is already deployed. Driving systems forecast where people and vehicles will be over the next few seconds. Equipment in a busy yard can be forecast the same way.

Other industries are earlier. Many products sold as twins are a static scan with a dashboard on top. That is useful for orientation. It cannot answer a question about the future.

The difference matters for money. A static twin saves site visits. A predictive twin can change decisions.

It can show where to stage crews before a storm. It can show which yard route stays safe while equipment moves.

The second kind is worth more. It also costs more to build and to keep current. Every sensor feed that keeps it live is a line in the budget.

The risk is paying for the second and receiving the first. Vendor language rarely separates them.

Two checks expose the gap. Does the twin change over time on its own? Does it respond when you intervene, by moving an asset or closing a gate?

The decision is which kind of twin a given use case needs. Many need only the static kind. I would recommend to pay for prediction only where a decision depends on it.

Ask the vendor one question. If we act on this site tomorrow, can the twin show what changes next?

Quick check

3 questions. Pick one answer each. Get them all right to mark this territory as checked.

1What does D-NeRF add to NeRF?

2What does object permanence mean in a 4D model?

3What separates a predictive twin from a static scan?

Key terms

Further reading

  1. D-NeRF: Neural Radiance Fields for Dynamic Scenes Albert Pumarola, Enric Corona, Gerard Pons-Moll, Francesc Moreno-Noguer, 2020 A canonical scene plus a time-conditioned deformation.
  2. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis Jonathon Luiten, Georgios Kopanas, Bastian Leibe, Deva Ramanan, 2023 Moving Gaussians that give dense 6-DOF tracking for free.
  3. 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering Guanjun Wu et al., 2023 Real-time dynamic scenes, 82 frames per second at 800 by 800.
  4. OccWorld: Learning a 3D Occupancy World Model for Autonomous Driving Wenzhao Zheng et al., 2023 Forecasting future 3D occupancy and the vehicle's own path.
  5. A Functional Taxonomy of World Models World Labs, 2026 Where 3D and 4D models fit.
  6. Marble World Labs, 2025 An explorable generated 3D world.

Search the atlas