Listen to a short story. Maya puts her keys in the kitchen drawer. She goes outside to play. While she is gone, her brother moves the keys into his jacket pocket.

Where are the keys now? You know they are in the jacket. Where will Maya look for them? You know she will open the drawer.

Nobody drew you a picture. You built a little picture of the kitchen from words alone. You kept track of two things at once.

One is where the keys really are. The other is where Maya thinks they are.

The chatbots you may have heard about learn in a strange way. They read a huge amount of text. Their only job is to guess the next word. Nobody tells them about kitchens or keys.

Yet scientists looked inside one of these models and found a surprise. One model only ever read the moves of a board game. Inside, it had built a picture of the board. It knew which piece sat on which square.

So these models can build little pictures of the world from words. The tricky part is how reliable those pictures are. Sometimes the picture is right.

Sometimes the model just sounds sure of itself and gets the keys wrong. That matters when a computer helper does real jobs for people. If it loses track of the keys, it sends someone to the wrong drawer.

Try this at home. Tell a grown-up a story where one toy moves three times. Then ask where the toy is now. Next, ask where the first person thinks it is.

Check whether the answers match your own picture.

A model trained only on next-token prediction can still carry a world model. The question is how far that model goes and when it breaks.

The strongest early evidence came from Othello-GPT (Li et al., 2023). A transformer saw only legal move sequences. It still built an internal representation of the board state.

Probes read the board out of its activations. Interventions on that representation changed the model's next-move predictions. That is causal evidence, beyond correlation.

Language can also serve as the state itself. An LLM can simulate an environment described in text and hold a belief state. It can predict the effect of an action in interactive fiction or a tool call.

The measured numbers are sobering. On text-based world simulation the best model scored 59.9% (Wang et al., 2024). That figure covers transitions with non-trivial changes. The authors call the model an unreliable simulator without further work.

Coherence is a separate test from accuracy. One team trained transformers on New York City taxi routes (Vafa et al., 2024). The models proposed valid turns almost every time.

Yet the street map implied by their outputs bore little resemblance to Manhattan. Performance fell apart once detours appeared.

A 2026 mechanistic study disputes that reading (Beckmann et al., 2026). It finds a faithful internal map and traces the failures to interference between superposed features. For a builder, both readings lead to the same rule. Good next-step predictions do not prove a usable state.

The failure modes are well documented:

• state tracking degrades over long sequences
• entity bindings get swapped
• physical commonsense breaks on unusual cases
• the model states a wrong belief with full confidence

Grounding is the remedy. Tie predictions to external state through tools, databases or simulators. Check each prediction against the next observation. Train with explicit state supervision where you can.

Take a customer operations case. A customer chats about moving. The agent starts service at the new address and sets up a payment arrangement.

Then a meter exchange posts at the old premise two days late. By turn forty, the model's picture binds the new meter to the old account.

I would recommend three controls. The agent reads the system of record before any write. It writes down the state it believes in a structured form. A checker compares that belief with the record and blocks on mismatch.

Asynchronous data makes this harder. The record itself may lag the field by hours. So the belief log needs a timestamp on each fact.

Measure state-tracking accuracy against ground truth as conversations lengthen. Track the belief-record mismatch rate per thousand actions. Count the blocked actions that were truly wrong.

This is the territory most enterprises meet first, often without knowing it. LLM agents already act for people. They answer customers and update records.

Every one of those actions rests on the model's private picture of the current state. Which account is this? What changed today? What did the customer already agree to?

That private picture is often right and sometimes confidently wrong. Research shows both sides. A model can track a game board it never saw. A similar model can lose track of who holds which item in a long story.

The consequence in a utility is concrete. A stale agent can set up a payment plan on a closed account. It can quote a balance that changed an hour ago.

Each error becomes a complaint and a correction cost. Some become a regulatory problem.

The practical rule is simple. Never let the model's picture be the only record of the state. Three controls follow from it:

• keep authoritative state outside the model
• make the agent read that state before it acts
• log what the agent believed at the moment it acted

Teams skip the third control most often. You need it when an auditor or a regulator asks why the agent acted.

These controls cost engineering time up front. They save far more in rework and dispute handling later. I would require them for any agent that writes to a system of record.

So ask one question before you approve an agent. When it acted, what did it believe the account looked like?