Why Is Capability Not the Same as Reliability?
Capability and reliability differ because a capable model's work can still be undermined by a vague objective, an unreproducible environment, or missing checks. SWE-agent shows why interface design deserves attention, but comparisons across models and years cannot isolate how much of a performance gain came from the harness alone.
A useful harness gives a model clear tools, visible state, and feedback from its own work. The SWE-agent project reported a 12.5% resolve rate on SWE-bench's real GitHub software issues with GPT-4 Turbo, compared with the original benchmark paper's 1.96% Claude 2 baseline. Both the model and the interface changed between those results, so the historical comparison motivates testing the harness but does not prove a sixfold gain from the interface alone.
SWE-bench results: different models and harnesses
Source: Jimenez et al. (arXiv:2310.06770); Yang et al. (arXiv:2405.15793)A harness leaves the model as smart as it was and builds a closed verification loop around it, so the capability the model already has actually gets realized.
The first rule of the doctrine follows from that. When an agent fails, attribute the failure to a harness layer before you blame the model, by asking five questions: was the task specified, was the context visible, was the environment reproducible, could success be verified, and was state persisted? In our experience, one of those five answers is almost always no.
What Are the Five Layers of an AI Agent Harness?
An agent harness has five layers: the task spec that defines done, the context the model can actually see, the environment it runs in, the verification that checks its work, and the state persisted between sessions. Chronic agent failures map to a weak layer, and verification is the highest-return layer of the five.
- Task spec: what exactly is being asked, with constraints and a definition of done the agent can check.
- Context: the instructions, documents, and tool definitions the model can actually see when it acts.
- Environment: pinned dependencies, a scripted setup, a startup path that works from a cold clone.
- Verification: executable feedback (lint, types, tests, end-to-end runs). It is the highest-return layer of the five.
- State: progress, decisions, and task status persisted in files, so work survives the end of a session.
Score each layer one to five for an agent system you run, then investigate the lowest score first. A weak layer gives you a concrete failure to test before paying for a model upgrade. The audit can start with a single failed task: record what the agent could see, which tools it used, and how its result was checked, then record the failure and repeat the test. We use this approach on workflow automation builds to make the next fix testable.
The benchmark record points the same way. Anthropic's engineers, writing up their own SWE-bench submission, noted that an agent's performance "can vary significantly based on this scaffolding, even when using the same underlying AI model." In that submission, the tool descriptions and error-proofing got more engineering attention than the prompt. The published results since the benchmark launched show models and harnesses improving together:
| Date | System | Benchmark | Resolved |
|---|---|---|---|
| Oct 2023 | Claude 2 with a retrieval harness | SWE-bench, 2,294 real GitHub issues | 1.96% |
| May 2024 | GPT-4 Turbo with SWE-agent | SWE-bench | 12.5% |
| Jan 2025 | Claude 3.5 Sonnet in Anthropic's agent scaffold | SWE-bench Verified, 500 human-validated tasks | 49% |
| Sep 2025 | Claude Sonnet 4.5 | SWE-bench Verified | 77.2% |
Why Is the Repo the Spec?
The repo is the spec because a fresh agent session needs the project rules and decisions it must follow, and verbal agreements or past chat threads may not be available. The repository gives the project overview, constraints, and verification commands a durable home between sessions. If a fresh session cannot continue from that context, the spec is incomplete.
If a rule exists only in an old conversation, a fresh agent session may never see it before it makes the same mistake. To test for this, start a new session with the repository context and ask the agent to continue. Missing project decisions and constraints show up quickly. This is one half of a larger problem we cover in our report on context engineering: the agent can only act on the information available when it acts.
How Do You Know an Agent's Work Is Done?
An agent's work is done when a command passes, not when it looks done. Every task gets a definition of done a machine can check, verified in three layers: static checks, then tests plus a real startup, then end-to-end runs. Unit tests alone miss boundary defects, and only an end-to-end pass proves the system works as a user experiences it.
The most valuable rule we enforce is that "looks fine" is not a state. Anthropic's guidance for its own coding agent opens with the same rule: give the agent a check it can run, because without one, "looks done" is the only signal it has. We cover the testing side in depth in our report on AI agent evaluation.
Why Should an Agent Work on One Task at a Time?
An agent should work on one task at a time because, left alone, agents open many workstreams and finish none of them. WIP=1 means exactly one task is active, and it reaches verified passing, with evidence, before the next one starts. Task status lives in a tracked file, and only verification output can promote work to passing.
Left to themselves, agents will open six workstreams and leave a repo full of half-built features that all claim progress. The tracked status file acts as a state machine. Agents are not allowed to promote their own work, because they systematically over-grade it, so the checker is never the worker.
Where Should Agent State Live?
State lives in files, never in session memory. A progress file is updated at the end of every working session, a decisions file records what was chosen and why, and git checkpoints mark every verified step. A fresh session should rebuild its bearings from files alone in about three minutes.
Context windows end and sessions crash. The agent that continues the work tomorrow remembers nothing about today unless today was written down, so we externalize state as a matter of policy. When a fresh session can recover its bearings in about three minutes, the work compounds across sessions instead of starting over every morning. The same reasoning is why our AI receptionist builds keep call outcomes, configuration, and handoff notes in durable records rather than in any one session's memory.
What Does a Clean Session Exit Look Like?
A clean exit means the build is green, every test passes including the ones that passed before, the progress file is current, debug cruft is deleted, and the startup path is verified. An agent that leaves a broken build has not finished its task, and it passes the failure on to the next session.
We count a clean handoff as part of done. An agent that leaves a red build for tomorrow has handed its failure to whoever opens the repo next, and by then it costs more to fix. The exit checklist runs every session, including the ones that went badly, because a failed attempt can still end with a clean stop.
Why Does the Harness Rot?
The harness rots because it is built around the weaknesses of current models, and models improve. Rules that prevent real failures today turn into pure ceremony within a year, so the harness itself gets audited like any other system: keep what still catches genuine failures, and delete what only adds process weight.
The doctrine has one caveat that keeps it from becoming dogma. A harness rule earns its keep by pointing at a real failure it recently prevented. A rule that cannot do that is bureaucracy, and it gets removed before it slows the work it was meant to protect.
Why Do We Hold This Line?
We hold this line because harness engineering is what gets an agent from a good demo to production. We run the doctrine on our own builds before asking any client to trust it, so every agent we deliver arrives with the spec, verification, and state that keep it reliable as the models underneath it improve.
Harness engineering is the foundation under our custom AI agent work, and clients get the agent together with the harness that keeps it reliable. Bigger models will keep arriving. The teams that benefit most will be the ones whose harnesses let an agent's capability show up in finished work.
