Failure compounds faster than per-step accuracy suggests

Most evaluations measure a single step. A lot of deployed systems run for tens or hundreds of steps, acting on their own previous output. This is what I found when I measured the difference.

What I did

I built 60 tasks in three families, each decomposable into a known number of required steps, and ran every task 40 times against each system with tool access held constant.

What came out

Per-step accuracy was high and similar across systems. End-to-end success was not, and the gap is mostly compounding: at 99% per step a 37-step task succeeds about 69% of the time, and at 97% per step about 32% of the time.

• Single-step accuracy stopped predicting end-to-end success at around ten steps.
• Error recovery mattered more than raw accuracy — systems that noticed their own mistakes and retried beat more accurate systems that did not.
• Run-to-run variance on one task exceeded the variance between systems, so small samples can flip a ranking.

What I take from it

If an evaluation is meant to inform a decision about deployment, it should test tasks about as long as the real ones and report the trial count. Testing a twenty-step workload with one-step benchmarks measures something, but not the thing that will break.

A number that does not say how many times it was measured is not a measurement.

The thing I keep writing in my own notes

Cite this page: Dingal AI Research. “Failure compounds faster than per-step accuracy suggests”. 29 July 2026.