Why Is Capability Not the Same as Reliability?
A capable model is not a reliable agent. The same model that fumbles a task with a vague objective, an unreproducible environment, and no way to check its work completes that task inside a tight harness. Public SWE-bench results show resolve rates multiplying roughly sixfold when only the interface around the model changed.
Today's models can do far more than most teams get out of them. The gap is not intelligence; it is everything around the intelligence. We have watched the harness flip happen on our own builds too many times to call it luck, and the public record agrees: the SWE-agent project multiplied frontier models' resolve rate on real GitHub issues roughly sixfold over the original benchmark baseline by changing nothing but the interface the model worked through.
Same benchmark, different harness: SWE-bench resolve rate
Source: Jimenez et al. (arXiv:2310.06770); Yang et al. (arXiv:2405.15793)A harness does not make the model smarter. It builds a closed verification loop around the model so the capability that already exists actually gets realized.
From that follows the first rule of the doctrine. When an agent fails, attribute the failure to a harness layer before you blame the model: was the task specified? Was the context visible? Was the environment reproducible? Could success be verified? Was state persisted? In our experience, one of those five answers is almost always no.
What Are the Five Layers of an AI Agent Harness?
An agent harness has five layers: the task spec that defines done, the context the model can actually see, the environment it runs in, the verification that checks its work, and the state persisted between sessions. Chronic agent failures map to a weak layer, and verification is the highest-return layer of the five.
- Task spec: what exactly is being asked, with constraints and a definition of done the agent can check.
- Context: the instructions, documents, and tool definitions the model can actually see when it acts.
- Environment: pinned dependencies, a scripted setup, a startup path that works from a cold clone.
- Verification: executable feedback, lint, types, tests, end-to-end runs. The highest-return layer of the five.
- State: progress, decisions, and task status persisted in files, so work survives the end of a session.
Score each layer one to five for any agent system you run, then fix the lowest score first. Every chronic agent problem we have diagnosed mapped to a weak layer, and not once has the cheapest fix been a model upgrade. The audit takes twenty minutes and tends to be uncomfortable, because the lowest score is usually a layer nobody designed at all. It just accreted. We run this audit on every workflow automation build before it goes live, for exactly that reason.
The benchmark record tells the same story at industrial scale. Anthropic's engineers, writing up their own SWE-bench submission, put it directly: an agent's performance "can vary significantly based on this scaffolding, even when using the same underlying AI model." Their tool descriptions and error-proofing got more engineering attention than the prompt. The published results since the benchmark launched are a history of models and harnesses improving together:
| Date | System | Benchmark | Resolved |
|---|---|---|---|
| Oct 2023 | Claude 2 with a retrieval harness | SWE-bench, 2,294 real GitHub issues | 1.96% |
| Jun 2024 | GPT-4 inside the SWE-agent interface | SWE-bench | 12.5% |
| Jan 2025 | Claude 3.5 Sonnet in Anthropic's agent scaffold | SWE-bench Verified, 500 human-validated tasks | 49% |
| Sep 2025 | Claude Sonnet 4.5 | SWE-bench Verified | 77.2% |
Why Is the Repo the Spec?
An agent only knows what is written in the repository. Verbal agreements, chat threads, and tribal knowledge are invisible to a fresh session. So the repo carries the project overview, the hard constraints, and above all the verification commands. If a brand-new session cannot continue the work from files alone, the spec is incomplete.
If a rule is not written in the repository, it does not exist for the agent. The test is brutal and simple: start a completely fresh session and ask the agent to continue the work. If it flounders, your spec lives in people's heads, and heads are not part of the harness. This is one half of a larger problem we cover in our report on context engineering: the agent can only act on what it can actually see when it acts.
How Do You Know an Agent's Work Is Done?
Done means the output of a passing command, never an impression. Every task gets a definition of done a machine can check, verified in three layers: static checks, then tests plus a real startup, then end-to-end runs. Unit tests alone miss boundary defects; only an end-to-end pass proves the system works as a user experiences it.
The single highest-leverage rule we enforce: "looks fine" is not a state. Anthropic's guidance for its own coding agent opens with the same rule: give the agent a check it can run, because without one, "looks done" is the only signal it has. We covered the testing side in depth in our report on AI agent evaluation.
Why Should an Agent Work on One Task at a Time?
Agents left alone open many workstreams and finish none. WIP=1 means exactly one task is active at a time, and it reaches verified passing, with evidence, before the next one starts. Task status lives in a tracked file, and the agent never promotes its own work to passing; only verification output does that.
Agents are eager. Left to themselves, they will open six workstreams and leave a repo full of half-built features that all claim progress. The tracked status file acts as a state machine, and the rule that the agent cannot self-promote exists because agents systematically over-grade their own work. The checker is never the worker.
Where Should Agent State Live?
State lives in files, never in session memory. A progress file is updated at the end of every working session, a decisions file records what was chosen and why, and git checkpoints mark every verified step. A fresh session should rebuild its bearings from files alone in about three minutes.
Context windows end. Sessions crash. The agent that continues the work tomorrow remembers nothing about today unless today was written down, so state is externalized as a matter of policy. That three-minute number is the difference between a system that compounds and one that starts over every morning. It is also why our AI receptionist builds keep call outcomes, configuration, and handoff notes in durable records rather than in any one session's memory.
What Does a Clean Session Exit Look Like?
A clean exit means the build is green, every test passes including the ones that passed before, the progress file is current, debug cruft is deleted, and the startup path is verified. An agent that leaves a broken build has not finished its task; it has exported the failure to the next session.
Clean handoff is part of done, not a courtesy. An agent that hands tomorrow a red build has charged its failure forward, with interest, to whoever opens the repo next. The exit checklist runs every session, including the ones that went badly, because a clean stop after a failed attempt is still a clean stop.
Why Does the Harness Rot?
Harnesses are built around the weaknesses of current models, and models improve. Rules that prevent real failures today turn into pure ceremony within a year. So the harness itself gets audited like any other system: keep what still catches genuine failures, and delete what only adds process weight.
One honest caveat keeps the doctrine from becoming dogma. A harness rule earns its keep by pointing at a real failure it recently prevented. When it cannot, it is bureaucracy wearing a safety vest, and it gets removed before it slows the work it was meant to protect.
Why Do We Hold This Line?
Harness engineering is the difference between AI that demos well and AI that ships. We run the doctrine on our own builds before asking any client to trust it, so every agent we deliver arrives with the spec, verification, and state that keep it honest as models keep improving underneath it.
It is the foundation under our custom AI agent work: clients do not just get an agent, they get the harness that keeps it honest. Bigger models will keep arriving. The teams that benefit most will be the ones whose harnesses let an agent's capability actually land.
