Fancy harnesses make coding agents more expensive but not better, study finds
Makers tout score gains from new agent harnesses like Claude Code or OpenCode. A study finds that claim doesn't hold up. Swapping harnesses barely moves results more than running the same setup twice, but costs can triple.
A coding agent pairs a language model with a harness, the system prompt, tools, and context management that make the model work inside a code repository. New harnesses ship every day, and leaderboards mix them freely, but two researchers at Shandong University measured how much the harness actually matters when you hold the model constant. The paper's title gives away the answer: "What does a harness buy? Mostly tokens."
Earlier comparisons were measuring noise
Previous studies report gaps of more than 20 points between harnesses, but those numbers almost always come from a single run, according to the authors. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI , for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin that falls well within that noise range. Harnesses are also often wired into the benchmark differently, making comparisons even less reliable.
Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode, and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and skips context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts, and automatic history compression.
The models were Qwen3.6 , Qwen3.8 , Deepseek-V4-Flash , GLM-5.3-Flash , and HY4-Preview. Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest tasks from SWE-bench Verified plus a pool of 447 more. Several setups ran twice to measure noise.
All runs happened offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task themselves. Claude models almost never did this. The researchers discarded those results and don't claim any maker trained on the test data, but the pattern was strong enough that they cut off web access for all remaining runs.
Swapping harnesses moves results about as much as a rerun
On the larger pool of 447 tasks, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models. Within a five-point band, they're statistically equivalent.
The researchers expected the biggest harness effect on the 45 hardest tasks. Instead, harnesses differ by two to five solved tasks per model, and a simple rerun of the same setup shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code does.
The flip rates tell the same story. Between two reruns, the median outcome changes on 13 percent of tasks. Between two different harnesses, it's also 13 percent. Swapping models pushes that to 22 percent. Which tasks a given harness wins changes from run to run, matching a coin flip. The harness gap behaves like a "fresh draw of the same noise," the authors write.
These small samples also limit what any comparison can detect. With 45 tasks, even a 13-point gap shows up only half the time. With 447 tasks, gaps of about five points become reliably detectable. Zhipu's claimed 2.4-point advantage falls below both thresholds, and so do many other advertised improvements.
All harnesses perform the same, except OpenCode falls behind
The only measurable harness effect is a steady deficit for OpenCode. On the large task pool, it trails the other two by up to nine points. About half of that gap traces back to an output limit. When a model exceeds this limit mid-step during reasoning, Claude Code prompts it to keep going. OpenCode lets the step end there, and the attempt usually fails.
Adding a Claude Code-style continuation prompt to OpenCode didn't help, because the same task rarely triggers the same overflow on a retry. For the rest of the gap, the authors have no explanation, only an observation: under OpenCode, models stop working much sooner. On locally run models, OpenCode also had a much tighter time limit than Claude Code, which the authors flag as an uncontrolled variable.
mini-SWE-agent runs into the context window because it lacks context management, mostly on tasks where the other harnesses also fail. OpenCode's search tools try to download a binary file on first use and break when offline.
System prompt and tool descriptions set the bill
With the same model, same tasks, and same price list, a single task can cost up to three times more depending on the harness. Tool outputs barely explain the gap. All three harnesses return similar amounts of text per call.
The real difference comes from the system prompt and tool descriptions. They get resent with every step and run about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, and under 1,000 for mini-SWE-agent. Claude Code also keeps the model's reasoning output in the conversation history, based on the researchers' reverse engineering.
After that first call, context grows at similar rates across all harnesses. The system prompt size sets the baseline cost, and the number of steps multiplies it. Which factor dominates depends on the model: With GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent needs so many extra steps that the two end up even. Lower prices for cached tokens dampen the gaps but don't change the ranking.
No home-field advantage, open questions around thinking
Deepseek's own harness scores no better with its own model than the other harnesses do, but costs the most. Claude Opus 5 delivered identical results even after the researchers removed tools from Claude Code. A year-old version solves a similar number of tasks at less than half the cost, because Extended Thinking wasn't on by default back then.
Thinking was the only case where the harness choice actually changed the outcome. Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points, while Claude Code and OpenCode stay within noise. HY4-Preview also loses eleven points in Claude Code without thinking. How much an agent depends on reasoning seems tied to the specific model-harness combination. The authors admit their test grid is too small to explain why.
The study relies mainly on one benchmark. A small cross-check on Terminal-Bench 2.1 shows the same pattern. The closed-source top model, Opus 5, ran only in Claude Code. A harness buys " fewer ways to lose a task, and a bill, not pass rate," the authors conclude. They recommend publishing a rerun of the same setup alongside every harness comparison. That's the only way to tell how much of a measured difference is just noise. Data and analysis scripts are available on GitHub .
More effort in the harness rarely pays off in other studies, either. In one, a simple Markdown file beat a complex skill system for AI coding agents. An earlier study already showed that the choice of framework can nearly triple costs for the same model. Harness design gets a lot of attention as a result. Nvidia recently cut nearly half its token costs for coding agents through better agent steering.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Read on for the full picture. Subscribe for hype-free coverage.
Full access to every article on THE DECODER
Join the comments and community discussions
A weekly AI news recap via mail
6x/year: "AI Radar" — deep dives on the AI topics that matter most
Daily AI news, always up to date
Covered by a team with 10+ years in AI