I think it’s really important, especially when you look at everything the tool does, from the execution loop and context management to feedback. The harness is basically the underlying source of truth.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
nojs 1 hours ago [-]
We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
snehesht 5 minutes ago [-]
I have been using jcode for past two weeks, honestly I feel its much better compared to Opencode.
Supermancho 4 hours ago [-]
The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc.
Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.
It is a starting point.
corv 3 minutes ago [-]
While sub-agents could theoretically stack turtles indefinitely, there’s limited room when going up the stack because someone will have to take responsibility beyond “git blame”, and that lands squarely with a person or legal entity.
corv 3 hours ago [-]
My own findings are in line with this research:
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
spike021 3 hours ago [-]
Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.
Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?
CaveTech 1 hours ago [-]
Your question is worded kind of confusingly, but all caching is handled on the inference layer, and by all major providers. In short, caching should work as long as you are sending requests to the same model and provider.
spike021 35 minutes ago [-]
Apologies. I guess my main question is just is there a benefit to routing directly to a provider through its own harness vs just sending chat requests to those providers using OpenRouter or similar as a proxy?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
x312 3 hours ago [-]
Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).
In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
ed_mercer 5 hours ago [-]
Does this extend to open models like GLM 5.3? This would mean that simply changing the harness to Pi reduces cost in half?
In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.
roywiggins 3 hours ago [-]
I've found the experience of using Pi with local models feels a lot snappier than both OpenCode or Claude Code.
lexicalmathical 3 hours ago [-]
If this is mostly because of the size of the system prompt, then perhaps in long horizon tasks the "tax" will be less obvious.
mikert89 3 hours ago [-]
As the model gets smarter, you need to tell it less
malnourish 2 hours ago [-]
Perhaps, but then the value of each thing you tell it goes to.
NewJazz 3 hours ago [-]
As the black box gets bigger, you need to stop expressing yourself even more.
jswelker 5 hours ago [-]
Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.
"Why pay the waste disposal tax? Dumping into the ocean is free!"
Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.
Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.
roywiggins 3 hours ago [-]
It's not hard to sandbox Pi without adding anything substantial to the actual size of the system prompt, which is what HarnessTax is mostly measuring. Wrapping it in nono.sh costs approximately zero tokens.
(I do think Pi should ship with more than zero builtin sandboxing though)
seanmcdirmid 3 hours ago [-]
You can sandbox with tool hooks, at least in antigravity, I think a lot of other harnesses support similar features. Sandboxing via the prompt is just dangerous and unreliable.
zackify 2 hours ago [-]
I have a couple hundred line TUI that just clones an LXD container and I get all my repos and infra isolated in 5s with pi having isolation to that container. Swap between any instance. Port forward to host all automatic.
Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.
KingOfMyRoom 60 minutes ago [-]
Do you have any links that would explain this?
I'm looking at changing my AI harness and am definitely interested in this
shshsjsj 4 hours ago [-]
[dead]
robinpie 4 hours ago [-]
claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product
topaz0 2 hours ago [-]
It's pretty great to not use, conversely.
shivrajnag 2 hours ago [-]
[dead]
nirmeetimthebes 49 minutes ago [-]
[dead]
appbeforelunch 3 hours ago [-]
[dead]
Rendered at 05:12:18 GMT+0000 (Coordinated Universal Time) with Vercel.
With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.
I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.
I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?
My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.
Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.
It is a starting point.
Having a coding harness is critical but the differences between them are overstated.
Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes
They’d all do the job - I just prefer to compartmentalize for access control.
Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly
Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?
As per your response it sounds like at least caching would happen for any provider regardless of the request's origin.
In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.
In the case of DeepSeek-V4-Flash-0731 on deepinfra.com there was little difference when both used high reasoning. In the case of that same model on together.ai there was a substantial difference between the two (high reasoning for both again). When using together.ai with Kilo Code the LLM was having a lot of trouble making successful edits. In some cases that meant a lot tries at using the tools and in others it worked around by running scripts. Meanwhile it used the tools from my harness just fine. I've specifically tried to make my tools easy for all of the open weight LLMs to use correctly. That was inspired by getting some errors from Kilo Code at the beginning of the year telling me that the model was having trouble and I should use a smarter model.
"Why pay the waste disposal tax? Dumping into the ocean is free!"
Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.
Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.
(I do think Pi should ship with more than zero builtin sandboxing though)
Funny to me how stuff is described these days. Seems like some people lack the full understanding of all these different systems and really overcomplicate their harness setups or use super bloated all in one tooling.
I'm looking at changing my AI harness and am definitely interested in this