Ai2 has released StateBench, a benchmark that tests how well language models keep track of entities and their changing states across long documents. State tracking underpins tasks such as reading code where variables are reassigned, or following a story where objects change hands. Some problems only require finding the latest update, while harder ones require following a chain of changes in which one missed step corrupts the final answer.
The benchmark has four difficulty levels, each allowing more complex update operations. The simplest uses values that never change, like a birth date. The next allows reassignment, like a thermostat setting. The third adds self-referential updates, like a running game score. The hardest allows one variable to take another's value, which Ai2 links to the S5 swap problem, where successive swaps must be tracked to locate an object. Researchers compared pretrained models including Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, along with smaller models of 275 million and 2.7 billion parameters. All were trained on the same data, none saw training contexts longer than 64,000 tokens, and most were tested on inputs well beyond that length. Almost all of the models handled the simplest level well. Ai2 reports that recurrent and hybrid models generalize much further than transformers when trained directly on these tasks. Frontier models, including those the report names as Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, still break down on harder forms of state tracking as inputs grow longer.
For Australian businesses and professionals, this is research rather than a product change, so no current tools, contracts or workflows need to change. The practical lesson is that long-document AI can fail quietly when a late update depends on earlier ones, such as contract amendments, revised financial models or reassigned variables in code. Teams using these tools for that kind of work should check outputs against source documents. The findings do not alter obligations under the Privacy Act 1988.
Key Points
StateBench tests how language models track changing entity states across long documents, from simple updates to long dependency chains.
The benchmark has four difficulty levels, from immutable assignment up to assignment by reference, which resembles the S5 swap problem.
Recurrent and hybrid models such as Olmo Hybrid, which adds Gated DeltaNet layers, generalize further than transformers when trained directly on these tasks.
Frontier models still break down on harder state-tracking tasks as inputs get longer.
For Australian businesses, the immediate impact is limited, but long-document outputs should be verified against sources.