Atlasof World Models
Start

T-04 Province I · The Founding Coast The Interior Survey 5 min 4 of 21

State-Space Models

How a model carries a compressed memory of the past forward through time.

State-Space Models: one fixed state, rewritten every reading Plain words: the scorecard, the plays and the replay. Architect: a fixed-size state x_t updated from input u_t, output y_t, filtering that uses readings so far, smoothing that also uses later readings, constant memory per step. CIO: the history, the memory that stays the same size, a constant cost per reading and the risk that a rare event is dropped quietly. T-04 · ONE SMALL STATE the scorecard the plays what you say a slow build-up you cannot see notes written live the replay fixes the notes One small card, rewritten after every play. xt, fixed size input ut, 96 reads a day yt slow mode, long memory filtering: uses reads up to t smoothing: uses reads after t too Constant compute and memory per step. Selective updates pick what stays. fixed-size memory the history, 35,000 reads a year answer rare slow risk flat cost per read live view after-action review Cost stays flat. Test that the rare event survives.
Mode

A long paper tape unrolling from a recording drum across a desk to a card marked with a cobalt tab.

In one sentence Carry one compact, ever-updated summary of the past forward, instead of re-reading everything every time.

Illustration of Sonu, a ten-year-old girl with a dark ponytail in a yellow shirt, holding a small notepad. Sonu, 10

Picture a long soccer game on a Saturday. You are the scorekeeper. You have one small card and a pencil.

You cannot write down every kick. The card is too small. So you keep only what matters.

You write the score and who has a yellow card. Maybe you note which player looks tired.

After every big play, you rub out a line and write a new one. The card stays the same size all game. It just keeps changing.

That little card is what a state-space model keeps. It is a short summary of the past that gets rewritten at every step. It carries just enough of the game to make sense of the next play.

A small card is also a risk. Say you rub out the yellow card to make room. Later the same player fouls again. You will be surprised, because your card forgot.

The trick is choosing what to keep. A good scorekeeper learns that yellow cards matter for the whole game. A goal from the first minute still counts at the end.

After the game, you can also watch the replay. Now you know how things turned out. You can fix your notes about the first half.

Scientists call that smoothing. Writing the card live, during the game, is called filtering.

Computers do this with long streams of numbers. One meter on one house sends 96 readings a day. Over a year that is more than 35,000 numbers.

The computer cannot reread all of them each time. It keeps a small card instead and updates it with each new reading.

Try this at school or at home. Watch one game or one TV episode with a grown-up. Keep a card with only three lines on it. Update it at every break.

At the end, ask the grown-up to quiz you about the start. What did your card remember? What did it lose?

Illustration of the Senior AI Architect, a person in glasses and a blue shirt holding a pen. Senior AI Architect

A state-space model carries a state forward through time. The new state is a function of the old state and the current input. The output is read from the state. Per-step compute and memory stay constant however long the sequence runs.

That property is why the lineage matters. Attention compares every token with every other token. Its cost grows with the square of the context length. A recurrent state pays a fixed price per step.

The classical form is linear. The Kalman filter (1960) estimates the state online from observations so far. That is filtering.

The Rauch, Tung and Striebel smoother (1965) revisits past estimates once later observations exist. Smoothing is more accurate and non-causal.

Deep learning revived the idea with structured recurrences. S4 (Gu, Goel and Ré, ICLR 2022) parameterized the state matrix for long-range memory. It solved the Path-X task of length 16,384, which all prior work had failed. It also generated 60 times faster than Transformers in their tests.

Mamba (Gu and Dao, 2023) made the state selective. The recurrence parameters depend on the input, so the model can choose what to keep. Mamba scales linearly in sequence length. Its authors report about 5 times the inference throughput of Transformers.

World models filter online for planning and often smooth offline for training. That gap is a common source of train and inference mismatch. A state trained with future information looks better in training than it can live. I would recommend to audit this before anything else.

Here is the feeder version, from my lab. Each hour the model sees 16 meter-cluster loads and the temperature. Air-conditioning load follows indoor heat build-up, a slow average of past temperatures. That variable is never in the observation.

A no-memory model predicted the next hour from the current hour and the action. A GRU carried a 32-number memory.

After 12 hours of history, the GRU's one-step error was 0.034 against 0.040. Nobody told it the heat variable existed. It carried some of it anyway.

Three failure modes deserve tests. A state too small forgets slow variables like heat build-up or a meter's drift. Selective forgetting can drop rare events, such as a brief overload. Smoothing leakage makes offline scores look better than live ones.

What to measure follows from that. Plot error by hours of history seen. Probe the state for slow variables you know exist. Then insert gaps in the readings and check recovery.

The utility fit is long memory at low cost. Equipment histories and multi-week outage restorations both run for thousands of steps. A fixed-size state keeps the cost flat as that history grows.

Illustration of My CIO, a grey-haired person in a purple jacket with arms folded. My CIO

State-space models answer a cost problem. Attention-based sequence models scale with the square of context length. That cost becomes prohibitive for anything that must remember hours or days.

A few thousand tokens is fine for a chat. It is far short for operational memory. One smart meter at 15-minute intervals makes 96 readings a day.

A feeder has many meters. An asset can run for decades.

The SSM lineage keeps a constant-size memory. Each new reading costs the same to process as the first. That makes long histories affordable to carry in production.

The applications with real long memory are familiar. A customer relationship spans years of bills and payment plans. An equipment history spans decades of inspections and repairs. A multi-week process such as storm restoration spans thousands of events.

Track this lineage for its constant-memory recurrence. A compelling generative demo says little about it.

There is a risk inside the saving. A compact memory has to choose what to keep. If it drops a rare event, the model forgets it quietly. A transformer with the full record in view would still see it.

So the decision is about which histories matter most. Where a rare past event drives today's risk, test the memory for that event. A transformer over a shorter window may also be enough for some uses. Pick by the length of memory the decision needs.

I would recommend to ask for one concrete test before funding. Plant a known event far back in the history. Then check whether the model still acts on it.

The question I would ask is this one. What does the model remember about this asset from six months ago? Can we see it?

Quick check

3 questions. Pick one answer each. Get them all right to mark this territory as checked.

1Why do state-space models suit very long histories?

2What is the difference between filtering and smoothing?

3In the feeder lab, what did the GRU memory pick up that no single hour showed?

Key terms

Further reading

  1. A New Approach to Linear Filtering and Prediction Problems R. E. Kalman, 1960 Filtering in its original form, a linear state carried forward one step at a time.
  2. Maximum Likelihood Estimates of Linear Dynamic Systems H. E. Rauch, F. Tung, C. T. Striebel, 1965 The smoother that revisits past states once later data arrives.
  3. Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu, Karan Goel, Christopher Ré, 2022 S4. The first model to solve Path-X at length 16,384.
  4. Mamba: Linear-Time Sequence Modeling with Selective State Spaces Albert Gu, Tri Dao, 2023 Input-dependent state updates with linear scaling in sequence length.
  5. Kalman Filter Tracking Dynamax Filtering and smoothing, worked end to end in a notebook.

Search the atlas