Atlasof World Models
Start

T-20 · World Models for Digital Agents

  1. Mission · Sonu, 10

    World Models for Digital Agents

    Before an agent clicks "submit", it should be able to say what the screen will look like afterwards.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Picture ordering a new book online with a grown-up. You find the book and press the big button that says Buy now.

    Before you press it, you already know what comes next. A thank-you page appears. An email arrives.

    Money leaves the family account. In a few days, a parcel lands at the door.

  3. Idea 2 of 4

    Keep in your head Picture ordering a new book online with a grown-up.

    You would hesitate if you were not sure. What if the button bought ten copies? What if it sent the book to Grandma's old house?

    Some computer helpers can now use websites for people. They click buttons and fill in forms. They do this very fast, without a grown-up beside them.

    Try this Close the screen. Tell someone at home what this card says, in your own words.

  4. Idea 3 of 4

    Keep in your head You would hesitate if you were not sure.

    These helpers need the same picture in their heads that you have. Before they click, they should know what the screen will show next. Builders call that a world model for a computer helper.

    With that picture, the helper can compare choices. Maybe the Save button is safer than the Send button right now. It can also spot a button that cannot be undone. Buying and deleting are like that.

  5. Idea 4 of 4

    Keep in your head These helpers need the same picture in their heads that you have.

    For those buttons, the helper should stop and ask a person first. A good helper knows the difference between a pencil and a pen. Pencil marks rub out. Pen marks stay.

    Try this at home. Next time you press a button, say out loud what will happen first. Try it with the microwave or a game. Then check whether you were right.

    Ask one more question each time. Can I undo this if I am wrong? If the answer is no, slow down and ask a grown-up.

    Try this Draw this card as one small picture. Use three lines or fewer.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1Why is predicting the effect of a click a world-model problem?

  8. Quick check 2 of 3

    2What does pass^k in τ-bench measure?

  9. Quick check 3 of 3

    3Which control fits an irreversible action such as a field disconnect?

  10. Recap

    What you just learned

    1. Picture ordering a new book online with a grown-up.
    2. You would hesitate if you were not sure.
    3. These helpers need the same picture in their heads that you have.
    4. For those buttons, the helper should stop and ask a person first.

  1. Mission · Senior AI Architect

    World Models for Digital Agents

    Before an agent clicks "submit", it should be able to say what the screen will look like afterwards.

    7 idea cards, 3 questions and a recap. About 7 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 7

    A digital agent acts inside software, from a web page to an enterprise system. It clicks and types. It also calls APIs.

    Predicting the effect of each action is a world-model problem. Here the world is an interface and the systems behind it.

    The benchmarks set the baseline. WebArena put agents in realistic, self-hosted web applications with verifiable goals (Zhou et al., 2023). At release the best GPT-4 agent completed 14.41% of tasks. Humans completed 78.24%.

  3. Idea 2 of 7

    Keep in your head A digital agent acts inside software, from a web page to an enterprise system.

    OSWorld moved the test to full desktop environments with 369 tasks (Xie et al., 2024). Humans completed over 72.36%. The best model reached 12.24%. Newer agents score higher, so treat these as starting points.

    A recurring failure is acting without anticipating consequences. Agents submit the wrong form, delete data or loop. One paper cites an agent repeatedly buying a non-refundable flight ticket (Chae et al., 2024).

    Try this Look away and say this card back in one sentence.

  4. Idea 3 of 7

    Keep in your head OSWorld moved the test to full desktop environments with 369 tasks (Xie et al., 2024).

    Its world-model-augmented agent predicts the outcome of each action first. It trains on short text descriptions of the key state change between steps.

    WebDreamer uses the LLM itself as the world model (Gu et al., 2024). It simulates the outcome of each candidate action, then picks one. It beat reactive baselines.

  5. Idea 4 of 7

    Keep in your head Its world-model-augmented agent predicts the outcome of each action first.

    In sandbox environments it matched tree search at four to five times the efficiency. This is model predictive control for interfaces.

    Reliability needs its own metric. τ-bench measures pass^k, success on all of k repeated trials (Yao et al., 2024). At release, a gpt-4o agent solved under half the tasks. Its pass^8 in retail fell below 25%.

    Try this Name one system you have shipped where this would break first.

  6. Idea 5 of 7

    Keep in your head In sandbox environments it matched tree search at four to five times the efficiency.

    Digital environments have advantages over physical ones. State can often be read exactly from the DOM, an API or a database. Environments can be snapshotted and replayed. So you can validate a predicted effect in a sandbox before you commit to production.

    That discipline comes from transactional systems.

  7. Idea 6 of 7

    Keep in your head Digital environments have advantages over physical ones.

    Take a disconnect for non-payment in a customer system. Before acting, the agent states its expected effect in structured form. One field order lands on tomorrow's route. The account status changes and a notice goes out.

    The agent then runs the action against a snapshot of the same account. A diff compares the predicted effect with the sandbox effect. Any extra change blocks the action. This step catches a medical-baseline flag or a pending payment.

    Try this Write down the one number on this card you would put on a dashboard.

  8. Idea 7 of 7

    Keep in your head Take a disconnect for non-payment in a customer system.

    The real action is irreversible in the field, because a crew truck rolls. So it goes through a human gate, with the prediction and the diff attached.

    Measure four things:

    • task success and pass^k on your own task set
    • prediction accuracy of expected effects against observed effects
    • irreversible actions attempted without a gate, which should be zero
    • the share of blocked actions a reviewer confirms as real catches
  9. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  10. Quick check 1 of 3

    1Why is predicting the effect of a click a world-model problem?

  11. Quick check 2 of 3

    2What does pass^k in τ-bench measure?

  12. Quick check 3 of 3

    3Which control fits an irreversible action such as a field disconnect?

  13. Recap

    What you just learned

    1. A digital agent acts inside software, from a web page to an enterprise system.
    2. OSWorld moved the test to full desktop environments with 369 tasks (Xie et al., 2024).
    3. Its world-model-augmented agent predicts the outcome of each action first.
    4. In sandbox environments it matched tree search at four to five times the efficiency.
    5. Digital environments have advantages over physical ones.
    6. Take a disconnect for non-payment in a customer system.
    7. The real action is irreversible in the field, because a crew truck rolls.

  1. Mission · My CIO

    World Models for Digital Agents

    Before an agent clicks "submit", it should be able to say what the screen will look like afterwards.

    4 idea cards, 3 questions and a recap. About 5 minutes. One idea per card, so read one, then look away and say it back.

    Arrow keys or swipe to move.

  2. Idea 1 of 4

    Digital agents are the world-model application most enterprises meet first. They fill forms and trigger workflows inside the systems you already run.

    Can the agent complete the task? That question comes second. The governing question is what the agent can change. Then ask who approves the changes that cannot be undone.

  3. Idea 2 of 4

    Keep in your head Digital agents are the world-model application most enterprises meet first.

    The public evidence argues for caution. On realistic web and desktop benchmarks, early agents finished few tasks. People finished most of the same tasks easily. Newer agents do better.

    Reliability across repeated runs is still the weak point. One benchmark checks whether an agent succeeds on every one of eight repeats. In retail, a strong model at release passed under a quarter of the time.

    Try this Name the one decision this card changes for you.

  4. Idea 3 of 4

    Keep in your head The public evidence argues for caution.

    In a utility the irreversible actions are easy to name. A disconnect order sends a crew. A refund moves money. A message to a customer cannot be recalled.

    Each mistake costs money and trust. Some end in a regulatory filing.

    I would recommend three controls for any agent that writes:

  5. Idea 4 of 4

    Keep in your head In a utility the irreversible actions are easy to name.

    • the agent states the expected effect of each consequential action before acting
    • it checks the actual effect afterwards and reports any difference
    • anything irreversible goes through a named human approver

    Digital systems make these controls cheaper than in the physical world. Engineers can copy the state, test the action and compare. That rehearsal step costs little next to one wrong disconnect.

    Ask for an inventory before a demo. List every action the agent can take. Mark each as reversible or not. Name the approver for each irreversible one.

    Then ask one question. Which actions can this agent take that we cannot undo?

    Try this Say what this card would cost you if a vendor got it wrong.

  6. Words to keep

    Try this Cover the list. Say each word and what it means, then check.

  7. Quick check 1 of 3

    1Why is predicting the effect of a click a world-model problem?

  8. Quick check 2 of 3

    2What does pass^k in τ-bench measure?

  9. Quick check 3 of 3

    3Which control fits an irreversible action such as a field disconnect?

  10. Recap

    What you just learned

    1. Digital agents are the world-model application most enterprises meet first.
    2. The public evidence argues for caution.
    3. In a utility the irreversible actions are easy to name.
    4. the agent states the expected effect of each consequential action before acting it checks the actual effect afterwards and reports any difference anything irreversible goes ...

Search the atlas