Harness economics · OpenBench
Same model. Different harness bill.
GPT-5.6 Sol
Four coding harnesses ran GPT-5.6 Sol on the same 15-task benchmark. Solvency reprices each harness’s proxy-measured token usage at the model’s current verified API rates.
OpenBench-only basis · never mixed with the general model leaderboard
Matched pass rate · different usage
Codex costs 3.77× Pi per solved task at the same 72.7% pass rate.
$1.37 vs $0.363 per solved task · point estimates, not a claim of statistical superiority
Lowest point estimate
Pi
$0.264 / attempt · $0.363 / solved
Price basis
Current API
$5 input · $0.50 cache read · $30 output per million tokens
Source: OpenBench harness benchmark (MIT) · verified 2026-08-23 · GPT-5.6 Sol prices verified 2026-08-21; source usage repriced, not an observed dollar bill
One model · one benchmark
Harness cost comparison
Ranked by cost per solved task. Only the harness changes; lower is better. Infrastructure and rate-limited failures are excluded exactly as OpenBench specifies.
| # | Harness | Version | Solved | Pass | $ / attempt | $ / solved | vs lowest |
|---|---|---|---|---|---|---|---|
| 1 | Pi | 0.80.10 | 32/44 | 72.7% | $0.264 | $0.363 | 1.00× |
| 2 | Claude Code | 2.1.214 | 35/45 | 77.8% | $0.327 | $0.420 | 1.16× |
| 3 | Grok Build | 0.2.103 (89c3d36fb6f1) | 37/45 | 82.2% | $0.491 | $0.597 | 1.64× |
| 4 | Codex | 0.144.5 | 32/44 | 72.7% | $0.997 | $1.37 | 3.77× |
Source: OpenBench harness benchmark (MIT) · verified 2026-08-23 · source usage repriced: measured tokens × current verified model prices; no Solvency loop or tier assumption
Measured by Solvency · own population
Eight arms, same model, run by Solvency
On 2026-08-26 Solvency ran GPT-5.6 Sol through eight arms of its own single-turn suite, 12 single-turn code tasks, deterministic hidden-test graders, temperature 0, 3 trials. Every arm passed every countable attempt, and the bill still spans 9.1×. Own task population: never comparable with the OpenBench table above.
| # | Arm | Access | Pass | $ / solved | vs lowest |
|---|---|---|---|---|---|
| 1 | Aider · aider 0.86.2 | metered provider (OpenRouter) | 100% | $0.0093 | 1.00× |
| 2 | API, no harness | metered API, single call, no scaffold | 100% | $0.0095 | 1.01× |
| 3 | Pi · 0.84.3 | metered provider (OpenRouter) | 100% | $0.0097 | 1.04× |
| 4 | Codex · codex-cli 0.150.0 | local subscription login | 100% | $0.0248 | 2.65× |
| 5 | Goose · 1.47.0 | metered provider (OpenRouter) | 100% | $0.0284 | 3.05× |
| 6 | OpenCode · 1.18.20 | metered provider (OpenRouter) | 100% | $0.0364 | 3.90× |
| 7 | Hermes Agent · Hermes Agent v0.20.5 (2026.8.19) | metered provider (OpenRouter) | 100% | $0.0717 | 7.68× |
| 8 | Cline · 3.0.60 | metered provider (OpenRouter) | 100% | $0.0851 | 9.12× |
Solvency Bench (first-party) · verified 2026-08-26 · harness-reported usage × verified catalog prices; cache writes at the uncached input rate (stated); the subscription arm's flat fee never enters the math. Full anatomy and findings in Research Note 02.
Cost basis
What this comparison means
This is neither a provider invoice nor Solvency’s modeled task tier. It is an independently reproducible reprice of source-observed usage.
Calculation
cost_per_attempt =
mean_uncached_input × current_input_price
+ mean_cache_read × current_cached_input_price
+ mean_output × current_output_price
cost_per_solved = cost_per_attempt / solve_rateTask tier, assumed loops, cache-hit controls, retry cap and frontier-efficiency assumptions cannot move these rows. Tests enforce that invariant.
Read with care
- Each arm has only 44 or 45 countable attempts, so the correctness intervals are wide.
- OpenBench reports subscription-backed access; Solvency applies current API token rates for comparability, not the subscription fee.
- Cursor and OpenCode are excluded because their usage is CLI-self-reported; Devin is excluded because its split usage is incomplete.
- The source flags Pi for a rerun after a surprising per-task swing. Treat the point estimate as provisional.