article --summary
The summary
A new benchmark tested 22 coding agent harness setups across five open-weight models, running them many thousands of times on terminal-based tasks. The authors found that the model itself explains most of the cost differences, but the harness still matters a great deal, and a setup that is cheapest for one model can be among the most expensive for another.
The study also looked at where the money goes. Roughly two-thirds of estimated spending landed on runs that failed verification or timed out, and the number of agent steps was the only factor that clearly tracked cost per successful task. The simpler harnesses performed well, but the study did not test their design choices separately, so it cannot confirm why they were cheaper.
Editorial note: This is a summary of an article first published by The New Stack. The full piece is on their site.