Why Is Capability Not the Same as Reliability?
A capable model is not a reliable agent. A vague objective, an unreproducible environment, or missing checks can undermine its work. SWE-agent shows why interface design deserves attention, but comparisons across models and years cannot isolate how much of a performance gain came from the harness alone.
A useful harness gives a model clear tools, visible state, and feedback from its own work. The SWE-agent project reported a 12.5% resolve rate on SWE-bench's real GitHub software issues with GPT-4 Turbo, compared with the original benchmark paper's 1.96% Claude 2 baseline. Both the model and the interface changed between those results. That historical comparison motivates testing the harness; it does not prove a sixfold gain from the interface alone.
SWE-bench results: different models and harnesses
Source: Jimenez et al. (arXiv:2310.06770); Yang et al. (arXiv:2405.15793)A harness does not make the model smarter. It builds a closed verification loop around the model so the capability that already exists actually gets realized.
From that follows the first rule of the doctrine. When an agent fails, attribute the failure to a harness layer before you blame the model: was the task specified? Was the context visible? Was the environment reproducible? Could success be verified? Was state persisted? In our experience, one of those five answers is almost always no.
What Are the Five Layers of an AI Agent Harness?
An agent harness has five layers: the task spec that defines done, the context the model can actually see, the environment it runs in, the verification that checks its work, and the state persisted between sessions. Chronic agent failures map to a weak layer, and verification is the highest-return layer of the five.
- Task spec: what exactly is being asked, with constraints and a definition of done the agent can check.
- Context: the instructions, documents, and tool definitions the model can actually see when it acts.
- Environment: pinned dependencies, a scripted setup, a startup path that works from a cold clone.
- Verification: executable feedback, lint, types, tests, end-to-end runs. The highest-return layer of the five.
- State: progress, decisions, and task status persisted in files, so work survives the end of a session.
Score each layer one to five for an agent system you run, then investigate the lowest score first. A weak layer gives you a concrete failure to test before paying for a model upgrade. The audit can start with a single failed task: record what the agent could see, which tools it used, and how its result was checked. Record the failure and repeat the test. We use this approach on workflow automation builds to make the next fix testable.
The benchmark record tells the same story at industrial scale. Anthropic's engineers, writing up their own SWE-bench submission, put it directly: an agent's performance "can vary significantly based on this scaffolding, even when using the same underlying AI model." Their tool descriptions and error-proofing got more engineering attention than the prompt. The published results since the benchmark launched are a history of models and harnesses improving together:
| Date | System | Benchmark | Resolved |
|---|---|---|---|
| Oct 2023 | Claude 2 with a retrieval harness | SWE-bench, 2,294 real GitHub issues | 1.96% |
| May 2024 | GPT-4 Turbo with SWE-agent | SWE-bench | 12.5% |
| Jan 2025 | Claude 3.5 Sonnet in Anthropic's agent scaffold | SWE-bench Verified, 500 human-validated tasks | 49% |
| Sep 2025 | Claude Sonnet 4.5 | SWE-bench Verified | 77.2% |
Why Is the Repo the Spec?
A fresh agent session needs access to the project rules and decisions it must follow. Verbal agreements and past chat threads may not be available. The repository is a durable home that survives between sessions for the project overview, constraints, and verification commands. If a fresh session cannot continue from that context, the spec is incomplete.
If a rule exists only in an old conversation, a fresh agent session may never see it before it makes the same mistake. The practical test is simple: start a new session with the repository context and ask the agent to continue. Missing project decisions and constraints become visible quickly. This is one half of a larger problem we cover in our report on context engineering: the agent can only act on the information available when it acts.
How Do You Know an Agent's Work Is Done?
Done means the output of a passing command, never an impression. Every task gets a definition of done a machine can check, verified in three layers: static checks, then tests plus a real startup, then end-to-end runs. Unit tests alone miss boundary defects; only an end-to-end pass proves the system works as a user experiences it.
The single highest-leverage rule we enforce: "looks fine" is not a state. Anthropic's guidance for its own coding agent opens with the same rule: give the agent a check it can run, because without one, "looks done" is the only signal it has. We covered the testing side in depth in our report on AI agent evaluation.
Why Should an Agent Work on One Task at a Time?
Agents left alone open many workstreams and finish none. WIP=1 means exactly one task is active at a time, and it reaches verified passing, with evidence, before the next one starts. Task status lives in a tracked file, and the agent never promotes its own work to passing; only verification output does that.
Agents are eager. Left to themselves, they will open six workstreams and leave a repo full of half-built features that all claim progress. The tracked status file acts as a state machine, and the rule that the agent cannot self-promote exists because agents systematically over-grade their own work. The checker is never the worker.
Where Should Agent State Live?
State lives in files, never in session memory. A progress file is updated at the end of every working session, a decisions file records what was chosen and why, and git checkpoints mark every verified step. A fresh session should rebuild its bearings from files alone in about three minutes.
Context windows end. Sessions crash. The agent that continues the work tomorrow remembers nothing about today unless today was written down, so state is externalized as a matter of policy. That three-minute number is the difference between a system that compounds and one that starts over every morning. It is also why our AI receptionist builds keep call outcomes, configuration, and handoff notes in durable records rather than in any one session's memory.
What Does a Clean Session Exit Look Like?
A clean exit means the build is green, every test passes including the ones that passed before, the progress file is current, debug cruft is deleted, and the startup path is verified. An agent that leaves a broken build has not finished its task; it has exported the failure to the next session.
Clean handoff is part of done, not a courtesy. An agent that hands tomorrow a red build has charged its failure forward, with interest, to whoever opens the repo next. The exit checklist runs every session, including the ones that went badly, because a clean stop after a failed attempt is still a clean stop.
Why Does the Harness Rot?
Harnesses are built around the weaknesses of current models, and models improve. Rules that prevent real failures today turn into pure ceremony within a year. So the harness itself gets audited like any other system: keep what still catches genuine failures, and delete what only adds process weight.
One honest caveat keeps the doctrine from becoming dogma. A harness rule earns its keep by pointing at a real failure it recently prevented. When it cannot, it is bureaucracy wearing a safety vest, and it gets removed before it slows the work it was meant to protect.
Why Do We Hold This Line?
Harness engineering is the difference between AI that demos well and AI that ships. We run the doctrine on our own builds before asking any client to trust it, so every agent we deliver arrives with the spec, verification, and state that keep it honest as models keep improving underneath it.
It is the foundation under our custom AI agent work: clients do not just get an agent, they get the harness that keeps it honest. Bigger models will keep arriving. The teams that benefit most will be the ones whose harnesses let an agent's capability actually land.
