A new decoy counting benchmark is drawing attention to a gap in today's frontier models: their ability to perceive an image does not reliably translate into correct reasoning about what it contains. The test centres on counting tasks in which misleading visual distractors, or decoys, sit alongside the target objects, forcing a model to separate what is actually there from what only looks like it belongs in the count.
Elorian's Glimpse is presented as an attempt to close that gap. Its approach draws on the idea of thinking with images, in which a model works with visual inputs during its reasoning, revisiting or examining regions of an image rather than making a single pass judgement. The Latent Space episode covers the benchmark findings and the design behind Glimpse, but the show notes give little measured detail, so any claimed improvement will need independent testing.
For Australian businesses and professionals, nothing changes immediately. Teams using AI to count, inspect or verify items in images, such as stock checks, quality control or document review, should test current vision tools on their own images before relying on them, because counting errors can go unnoticed. Any workflow that sends images containing personal information to an AI service still falls under the Privacy Act 1988 and the Australian Privacy Principles, a check that matters more day to day than the benchmark itself. Trying a new visual reasoning tool is only worth the time for teams with vision-heavy tasks. For everyone else, this is a research signal rather than a reason to change tools.
Key Points
A new decoy counting benchmark shows that frontier models can perceive images yet still miscount when distractors are present.
The gap is framed as a seeing-versus-reasoning problem, not only a perception problem.
Elorian's Glimpse uses thinking with images, revisiting visual inputs during reasoning, to try to close the gap.
The show notes offer little measured data, so claimed gains need independent verification.
For Australian teams, the practical step is testing vision tools on their own images before relying on their counts.