Methodology
Solvency publishes one number that does not appear on any pricing page: what it costs to get a task finished.
cost_per_attempt = loops × (tokens_in × input_price + tokens_out × output_price)
cost_per_solved_task = cost_per_attempt / pass_rateTwo cost bases, never mixed
Where a benchmark publishes a per-task cost it actually observed, that figure is used directly and no Solvency assumption touches it — no loop count, no per-loop token estimate, no efficiency multiplier. Where only a pass rate is published, cost comes from a loop model that is an assumption, and is labelled as one. The two are reported separately and are never averaged.
Retry variants
Three are computed. Reports lead with naive (cost / p). With no human-takeover cost, the rigorous truncated-geometric variant reduces exactly to it. The capped variant understates cost for weak models, because capping attempts while booking nothing for never-solved tasks makes failure look cheap.
Rules
- Prices are verified against the provider's own pricing page on the date recorded. Prices recalled from memory are never used.
- Missing is missing. A number that does not exist is reported as missing, never estimated, and a missing cache price is never treated as zero.
- Every modelled parameter is labelled an assumption, carries its provenance, and is adjustable.
- Benchmark figures are third-party, cited and linked, and are not redistributed.
What this does not measure
A METR randomised controlled trial found experienced open-source developers were19% slower using early-2025 AI tools while believing they were20% faster. Cost per solved task measures benchmark cost. It is not a productivity measure, and the two must not be conflated.