Benchmark results
AutoMode's measured cost and pass rates against Claude Code on Opus, Sonnet high and effortless, on mixed sessions and 30 coding tasks, with its limits.
In short
AutoMode was measured against plain Claude Code on Opus, plain Sonnet high and the effortless router on the same five mixed sessions. Each session ran in one continuing conversation on a real repository: a question, a coding task, another question, then a harder task. Hidden tests graded every task. All four arms passed all 20 steps. AutoMode cost $2.01, Sonnet high $2.14, effortless $3.87 and Opus $4.14, so AutoMode was 51% cheaper than Opus. Most of the gap comes from questions: Haiku answered ten of them for $0.08, against $0.50 on Opus. On 30 single coding tasks, AutoMode passed 29 at $0.275 per passed task, against 28 at $0.499 for Opus, which is 45% less. Plain Sonnet high passed all 30 at $0.240, so on long coding tasks alone AutoMode costs about 15% more than Sonnet high. Costs are list-price equivalents, judges and subagents included.
In a typical session, AutoMode costs half as much as Claude Code on Opus, 48% less than effortless, and less than Sonnet high, at the same quality.
Mixed sessions
A typical session mixes quick questions with real coding work. Each of the 5 sessions runs in one continuing conversation, on a real repository: a question, a task, another question, then a harder task. Every task is graded by its own hidden tests, and every answer by the facts it must name.
| 5 mixed sessions, 20 steps | Passed | Questions | Coding tasks | Total | vs Opus |
|---|---|---|---|---|---|
| AutoMode | 20/20 | $0.08 | $1.93 | $2.01 | −51% |
| Plain Sonnet high | 20/20 | $0.24 | $1.90 | $2.14 | −48% |
| effortless, on Opus | 20/20 | $0.43 | $3.44 | $3.87 | −7% |
| Plain Opus (Claude Code default) | 20/20 | $0.50 | $3.64 | $4.14 | — |
Questions are where AutoMode pulls ahead. Haiku answers them for a third of Sonnet's cost and a sixth of Opus's, and the session stays warm for the next task. Coding tasks run on Sonnet at the effort a judge picks, at Sonnet's own price.
Estimate your own saving
The cost calculator multiplies your weekly mix of questions and tasks by these per-step costs.
Single coding tasks
On single coding tasks, AutoMode is 45% cheaper than Opus and passes one more task.
| 30 real coding tasks | Passed | Cost per passed task |
|---|---|---|
| AutoMode | 29/30 | $0.275 |
| Plain Opus | 28/30 | $0.499 |
| Plain Sonnet high | 30/30 | $0.240 |
The cost ratio against Opus is 0.551, with a 90% interval of 0.511 to 0.591. That clears the ship rule fixed before the run: no task lost and a ratio of 0.85 or less. The one task AutoMode missed, sqlglot, plain Sonnet high passed only 1 time in 3, and plain Opus failed it too. Against Sonnet high, AutoMode costs about 15% more on single coding tasks. The Sonnet high rows come from an earlier run on the same tasks and tests.
Where AutoMode does not win
No router wins every workload, so here is ours.
- Long single coding tasks with no questions. AutoMode runs them on Sonnet, so it costs about what Sonnet high costs, plus its judges and checks. If all you ever do is long coding tasks, and you already run Sonnet high, AutoMode saves you little.
- Haiku on hard work. A Haiku-first mode (
haikuFirst: t3, withrecovery: rescueandreview: observed) cost 44% less than Sonnet high on 14 tasks: $1.74 against $3.09. It failed 1 of them, mathjs-nullish, which Sonnet high passed. So it ships off. Turn it on if your work is mostly small and well tested.
Judge accuracy
node bench/judge-bench-effortless.mts measures how often the judge picks a sensible effort, on labeled cases. One run got 74 of 75 cases right, and 20 of 20 held-out cases. It measures the effort label only.
Last updated on
Report a vulnerability
How to report a security problem in AutoMode privately, what counts as in scope, which versions get fixes, and how fast the maintainers reply.
How the benchmark works
How AutoMode's benchmark works: pinned public repositories, hidden tests, list-price costs with judges included, and rules fixed before each run.