State-Space Models

Try to summarize a movie in one sentence after every scene, using only your last summary plus the new scene. By the end, your one-sentence summary has to somehow still capture everything that mattered from two hours ago. That running, ever-updated summary is what a state-space model computes — a compact memory that gets rewritten at every step, carrying forward just enough of the past to make sense of the present.

A state-space model (SSM) defines a state h_t updated by h_t = f(h_{t-1}, x_t), with an output read out as y_t = g(h_t). Classical linear SSMs (and their deep-learning descendants — S4, Mamba, and other selective-state architectures) are prized because, unlike attention, their per-step compute and memory are constant regardless of sequence length, while still supporting long-range dependencies through structured, often diagonalizable, recurrence matrices.

Filtering estimates the current state from observations up to now; smoothing revisits past state estimates once later observations are available, producing a more accurate — but non-causal — reconstruction. World models typically filter online (for planning) and smooth offline (for training), and the gap between those two modes is a recurring source of train/inference mismatch worth auditing.

State-space models are the practical answer to a cost problem: attention-based sequence models scale quadratically with context length, which becomes prohibitive for anything that needs to remember hours or days rather than a few thousand tokens. For applications with genuinely long operational memory — a customer relationship, an equipment history, a multi-week process — the SSM lineage is worth tracking specifically for its constant-memory recurrence, independent of how compelling any single generative demo looks.