Atlasof World Models
Start

T-14 · Spatial World Models II: 4D Dynamics & Interaction

  1. Mission · Sonu, 10

    Spatial World Models II: 4D Dynamics & Interaction

    A 3D map is a photo of a place. A 4D model is a film of it that you can step into.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    A map of a park shows you the paths and the pond. It shows where every tree stands. It does not show the dog running across the grass. It does not show the swings moving.

    For that you need time.

    A 4D world model is a 3D place that changes over time. People walk. Gates open. A ball rolls under a bench.

  3. Idea 2 of 4

    Keep in your head A map of a park shows you the paths and the pond.

    Here is the clever part. When the ball is under the bench, you cannot see it. The model still knows it is there.

    Grown-ups call this object permanence. You learned it so long ago that you never notice you use it. Peekaboo is a game about it.

    Try this Find the words "object permanence" on this card. Say what they mean in your own words.

  4. Idea 3 of 4

    Keep in your head Here is the clever part.

    The model also follows each thing as it moves. The dog at the gate is the same dog at the pond a moment later. If two dogs cross paths, the model should not mix them up.

    That is harder than it sounds. Two brown dogs look alike from far away. The model has to remember which way each one was running.

  5. Idea 4 of 4

    Keep in your head The model also follows each thing as it moves.

    The hardest part is pushing. Imagine you could reach into the model and push a swing. It should swing.

    Push a bench and it should slide or stay put. That depends on how heavy it is. A model that knows those things is more than a pretty picture. It is a world you can think with.

    Try this at the playground. Pick one person or one dog. Follow them with your eyes for a minute.

    When they go behind something, guess where they will come out. Were you right? What helped you guess?

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1What does D-NeRF add to NeRF?

  8. Quick check 2 of 3

    2What does object permanence mean in a 4D model?

  9. Quick check 3 of 3

    3What separates a predictive twin from a static scan?

  10. Recap

    What you just learned

    1. A map of a park shows you the paths and the pond.
    2. Here is the clever part.
    3. The model also follows each thing as it moves.
    4. The hardest part is pushing.

  1. Mission · Senior AI Architect

    Spatial World Models II: 4D Dynamics & Interaction

    A 3D map is a photo of a place. A 4D model is a film of it that you can step into.

    7 idea cards, 3 questions and a recap. About 7 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 7

    Dynamic scenes add a time axis to the 3D methods of the previous territory. D-NeRF (Pumarola et al., 2020) keeps a canonical scene. It learns a time-conditioned deformation from that scene into each moment. It renders new views of moving objects from a single moving camera.

  3. Idea 2 of 7

    Keep in your head Dynamic scenes add a time axis to the 3D methods of the previous territory.

    Gaussian methods move the primitives themselves. Dynamic 3D Gaussians (Luiten et al., 2023) let each Gaussian move and rotate over time. Its colour, opacity and size stay fixed.

    Local rigidity priors keep neighbours moving together. Dense 6-DOF tracking falls out of the reconstruction with no flow input.

    Try this Find "tracking" on this card. Define it in one line without looking.

  4. Idea 3 of 7

    Keep in your head Gaussian methods move the primitives themselves.

    4D Gaussian Splatting (Wu et al., 2023) predicts deformations with a small network. It renders at 82 frames per second at 800 by 800 on an RTX 3090.

    Three representations do most of the work:

    • Scene flow estimates 3D motion per point. - Long-term tracking keeps identities through occlusion. - Dynamic occupancy forecasts which space will be filled.
  5. Idea 4 of 7

    Keep in your head 4D Gaussian Splatting (Wu et al., 2023) predicts deformations with a small network.

    Dynamic occupancy is the workhorse in autonomous driving. OccWorld (Zheng et al., 2023) tokenizes 3D occupancy. A GPT-style transformer then predicts future scenes. It predicts the vehicle's own trajectory alongside them.

    Occupancy has practical advantages. It is cheap to get from sparse LiDAR. It works with cameras too. It describes shape at a finer grain than bounding boxes.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 7

    Keep in your head Dynamic occupancy is the workhorse in autonomous driving.

    Interaction is the frontier. Predicting how a scene responds to pushing, opening or grasping needs more than appearance.

    The representation must know where each object ends and how rigid it is. It must also model contact. This is where spatial models meet learned physics and robot learning.

  7. Idea 6 of 7

    Keep in your head Interaction is the frontier.

    Take storm response. Drone and truck cameras capture a damaged feeder over two days. A 4D model holds the scene as it changes.

    A leaning pole tilts further. A crew clears a fallen tree. A section of line goes back up.

    Planners can scrub through time and see what changed between passes. The tracking problem is real. The same pole must keep one identity from pass to pass. The angle and the light change every time.

    Try this Find "tracking" on this card. Define it in one line without looking.

  8. Idea 7 of 7

    Keep in your head Take storm response.

    A wrong identity merge is costly. Two poles become one in the record. A crew then drives to the wrong span.

    Interaction matters here too. A planner may ask what happens if a crew drops a tree to one side. Today that answer comes from people, with the model as the shared picture. Learned dynamics for that question are still research.

    Measure these three:

    • Identity accuracy through occlusion and across capture passes. - Occupancy forecast error at fixed horizons. - Response to an intervention, compared with what really happened.
  9. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  10. Quick check 1 of 3

    1What does D-NeRF add to NeRF?

  11. Quick check 2 of 3

    2What does object permanence mean in a 4D model?

  12. Quick check 3 of 3

    3What separates a predictive twin from a static scan?

  13. Recap

    What you just learned

    1. Dynamic scenes add a time axis to the 3D methods of the previous territory.
    2. Gaussian methods move the primitives themselves.
    3. 4D Gaussian Splatting (Wu et al., 2023) predicts deformations with a small network.
    4. Dynamic occupancy is the workhorse in autonomous driving.
    5. Interaction is the frontier.
    6. Take storm response.
    7. A wrong identity merge is costly.

  1. Mission · My CIO

    Spatial World Models II: 4D Dynamics & Interaction

    A 3D map is a photo of a place. A 4D model is a film of it that you can step into.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    4D models are the bridge between two kinds of digital twin. The first kind looks right. The second kind predicts.

    A 3D scan shows where things are. A 4D model shows how they move.

    Some of this is already deployed. Driving systems forecast where people and vehicles will be over the next few seconds. Equipment in a busy yard can be forecast the same way.

  3. Idea 2 of 4

    Keep in your head 4D models are the bridge between two kinds of digital twin.

    Other industries are earlier. Many products sold as twins are a static scan with a dashboard on top. That is useful for orientation. It cannot answer a question about the future.

    The difference matters for money. A static twin saves site visits. A predictive twin can change decisions.

    Try this Name the one decision this card changes for you.

  4. Idea 3 of 4

    Keep in your head Other industries are earlier.

    It can show where to stage crews before a storm. It can show which yard route stays safe while equipment moves.

    The second kind is worth more. It also costs more to build and to keep current. Every sensor feed that keeps it live is a line in the budget.

  5. Idea 4 of 4

    Keep in your head It can show where to stage crews before a storm.

    The risk is paying for the second and receiving the first. Vendor language rarely separates them.

    Two checks expose the gap. Does the twin change over time on its own? Does it respond when you intervene, by moving an asset or closing a gate?

    The decision is which kind of twin a given use case needs. Many need only the static kind. I would recommend to pay for prediction only where a decision depends on it.

    Ask the vendor one question. If we act on this site tomorrow, can the twin show what changes next?

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1What does D-NeRF add to NeRF?

  8. Quick check 2 of 3

    2What does object permanence mean in a 4D model?

  9. Quick check 3 of 3

    3What separates a predictive twin from a static scan?

  10. Recap

    What you just learned

    1. 4D models are the bridge between two kinds of digital twin.
    2. Other industries are earlier.
    3. It can show where to stage crews before a storm.
    4. The risk is paying for the second and receiving the first.

Search the atlas