Any agent can be made to look good. Count the actions it completed, ignore the ones the user immediately undid, treat every time it declined as a failure of capability rather than an act of honesty, and you will produce a chart that climbs.
So before measuring anything I wrote down the ways this measurement could lie — as a numbered list, in a spec section titled “every one of these is a mistake already made here.” Not hypothetical failure modes. Ones already committed in this codebase.
The unit is an attempt, and refusal is a result
Every graded event resolves to one of eight outcomes. The two that matter:
refused-honestly— the system said it couldn’t, and that was true.ungradeable— we genuinely cannot tell — never a synonym for success.
A refusal is not a failure. If an agent declines because the thing genuinely cannot be done, and says so, that is the behaviour I want, and a scoring system that buries it in a failure rate will train me to build an agent that lies rather than declines.
ungradeable carries a required reason. An unexplained one is the exact
silent failure the subsystem exists to prevent.
Four of the eight rules
A refusal is never a failure. Asks and errors stay separate, so an arm that asks more does not look like an arm that errors more.
An undone action is retroactively wrong. Counting the commit and ignoring the undo is how a system reports success at the moment it did damage.
Never derive an outcome from model self-report. No field may come from what
the model said it did — because an action here was once acknowledged
success: true while the document was byte-identical.
Sample size travels with every number. No rate is reported without its n,
and below a stated threshold a comparison returns underpowered by name,
never as “no effect”. Reporting no effect from four sessions is the most likely
way this apparatus would mislead its own author.
The doctrine as an assertion
The rules are only worth the tests that pin them. The keystone one: a session consisting entirely of honest refusals must score 0% failure, not 100%. Every derived rate also gets a direction test — construct a session where the rate must go up, and assert that it does — because a correction rate here was once silently inverted by widening its denominator, so an arm that provoked more refusals scored as needing less correction.
There is no statistical inference: no p-values, no confidence intervals.
underpowered is a crude threshold on n, deliberately, because a crude
honest signal beats a sophisticated one nobody can audit.
What the humans found
Sittings join the same corpus, and the prose stays: a person’s judgement about why something failed is the most valuable field in the whole record and is not flattened into an enum.
The last frame is the one I would point at. A test was left pending because the fix for it had been verified on a neighbouring code path rather than driven directly — “that is inference”. It would have been trivially easy, and completely wrong, to mark it passed.
What this does not claim
This grades whether the system did what was asked. It does not grade whether what it produced was any good.
And the corpus is small: two documented human sittings, plus development sessions. There is no population here, no randomisation — which arm a sitting runs is a human decision — and no finding I would present as a result. The instrument is built and tested; it has not yet been pointed at enough people to say anything. That is the honest state, and by rule 8 it travels with every number this thing will ever report.