How the benchmark works

Costs are list-price equivalents of every paid request, judges and subagents included. Each arm ran the same tasks and hidden tests, in sessions launched on Opus, as most people run Claude Code. Every decision rule was committed before the first row of its run. One repetition per arm makes this a release signal, not a general law.

A taskCopy Anchor Link

A task names a public repository, a pinned base commit and a pinned reference commit, plus hidden tests and a test command. See bench/tasks/README.md.

Three tracks that never mixCopy Anchor Link

Hidden tests are the primary result. Judge scores are secondary. A report never collapses quality, cost and time into one score.

What keeps a run honestCopy Anchor Link

• The workspace holds only the tree at base, in a new repository. The reference commit is out of reach through git.
• Hidden tests go in after the agent's change is saved, so the grader never sees them.
• Each run_id hashes the arm, the plugin code that arm runs, the prompt and the task, so a resume never reuses a row from another setup.
• A usage limit, a failed setup or a test command that never returns is not a result. Those rows stay out of the numbers.
• A run whose transcript shows the reference commit, the repository cache or another project's memory is left out.
• Intervals resample tasks, then repeats, stratified by cohort. Under 30 tasks, a report says pilot.

Reproduce the decisionCopy Anchor Link

The release decision needs no model call:

node bench/decide.mts bench/results/v6-s2-release.jsonl --policy bench/experiments/policies/v6c-opus-vs-opus.json

Read the v6 report, the rows and the harness guide.