StateBench: LLMs lose track of state in long documents

What was released?

The Allen Institute for AI (Ai2) released StateBench (Hugging Face / Ai2), a benchmark for how well language models track the evolving state of entities in long documents. It ships with a tech report, a dataset and code.

In code terms: what a variable holds after many assignments. In everyday terms: who holds the knife at the end of a mystery novel after it changed hands three times.

Four difficulty levels

• R-trivial: the value never changes, so the model only needs the first assignment (let x = 4).
• Aperiodic: reassignment by value, so finding the latest value is enough (x = 4, later x = 2).
• Soluble: self-referential updates that must be carried forward (x = x + 5).
• General finite: assignment by reference, such as x = y or swaps. The classic ball under five shuffled cups belongs here.

The levels are grounded in Krohn–Rhodes theory. All four versions are derived from the same sequence with the same number of variables, only the assignment type changes, so the levels compare cleanly.

What they found

Four frontier models were tested: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All did well on the first two levels. On soluble and general finite problems, accuracy fell sharply as inputs grew, dropping to 13% or lower on the longest general finite problems.

Source: Ai2. Accuracy of frontier models across the four StateBench levels versus input length.

Open pretrained models (Olmo 3, Olmo Hybrid and smaller variants) showed a similar pattern: fine on r-trivial, degrading on aperiodic with length, and at chance on general finite. Format mattered too: the same problem framed as spreadsheet edits went far worse than in RULER style.

The most telling part is direct training. Ai2 trained 275M-parameter models from scratch on sequences of up to 256 assignments and tested them on up to 64,000. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels. Transformers learned the easier levels but degraded on longer sequences and never learned the hardest one.

Why it matters

If you build on long context (code analysis, agent sessions, long contracts), this is a concrete warning: looking up the latest value works, but following a chain of changes over large inputs is not reliable yet. StateBench also gives a sharper yardstick for comparing recurrent and hybrid architectures. An open gap: the framework assumes a fixed set of entities, while real tasks keep introducing new ones.