A new analysis from IBM Research reveals a critical gap in how AI agent reliability is measured. While a ReAct agent backed by GPT-4.1 reports 77.4% average success on benchmark tasks, it succeeds on all five repeated runs for only 53.0% of those same tasks—a 24.4 percentage point consistency gap. This discrepancy exposes a fundamental problem: standard industry benchmarks report mean performance, which hides a costly unreliability problem in production environments where mission-critical tasks like financial reconciliation or contract validation demand consistent execution. The inconsistency stems from how language model probability distributions shape decision points in agent workflows. Some decisions emerge from sharp probability distributions where one token choice dominates, producing consistent behavior run after run. Others emerge from flat distributions where multiple token choices carry similar probability mass, causing the agent to "flip" between different actions on identical requests. These flip-prone steps compound across a trajectory of dozens of decisions, creating volatility that no amount of model capability or decoding optimization can fully eliminate. IBM's solution combines a diagnostic tool called the Consistency Analyzer with a new guideline type that targets reliability directly. By resampling decision points from recorded agent trajectories, the system identifies unstable steps and generates consistency guidelines that stabilize them at inference time. The result: the 24.4-point consistency gap shrinks to 12.0 points without sacrificing average accuracy, bringing practical production reliability in line with reported performance.