Researchers at HumeAI have discovered that several of the highest-performing open-source speech recognition models are optimizing for benchmark tests rather than improving at real-world transcription tasks. By evaluating 11 widely-used ASR models, the team found that leading systems reproduced errors from benchmark datasets like VoxPopuli even when the audio contradicted the reference transcripts. The models appeared to detect subtle acoustic cues indicating which benchmark they were being tested on, allowing them to produce the expected answer rather than accurately transcribing what was actually said. The research introduces three new tests to measure "benchmark optimization" or "benchmaxxing"—a phenomenon where public benchmarks become gamed because models learn benchmark-specific patterns rather than underlying skills. In one case study, six of eleven models omitted the phrase "Thank you" from a VoxPopuli clip to match the benchmark's erroneous reference transcript, even though the audio clearly contained the phrase. When the same content was re-recorded with different voices or new speakers, most models switched to transcribing what they actually heard, suggesting they had learned to identify and conform to specific benchmark characteristics rather than improving their actual transcription ability. To address this measurement gap, Hugging Face has introduced held-out test sets across Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard. The findings underscore a broader challenge in AI evaluation: public benchmark scores may overstate real-world performance and mask fundamental limitations in model reliability, naturalness, and contextual appropriateness for practical deployment.