Why AI Agents Fail on Tasks They Can Actually Do

Why Does a Capable Model Still Fail at Real Work?

A model's raw capability and an agent's reliability are two different things. The same model that fails a task with a vague brief, no real environment, and no way to check its own work finishes that exact task once those three things exist. The gap is rarely intelligence.

Capability is what a model can do in principle. Reliability is whether it does that thing correctly, on your data, in your environment, without a person watching over its shoulder. Benchmarks measure the first. Production measures the second, and the two numbers are not close. The strongest public coding agents resolve roughly half of hand-picked SWE-bench Verified issues, tasks that already come with clean descriptions and a ready-made test. Turn a real engineer loose on a vague ticket with no tests and undocumented conventions, and the success rate drops hard, not because the model got dumber, but because everything that used to compensate for ambiguity, a coworker's context, a shared understanding of the codebase, disappeared.

We build custom AI agents for a living, and we've watched the same experiment play out on our own work more times than we'd like to admit. Same model, same prompt, wildly different outcomes, because one run had a written definition of done and the other didn't. Anthropic ran a version of this test formally: one prompt, one model, a bare single-agent run against a run wrapped in a planner, generator, and evaluator loop. The bare run finished in 20 minutes for $9 and shipped a broken game. The harnessed run took six hours and $200 and shipped a fully playable one. Nothing about the model changed between those two runs.

What Are the Five Subsystems That Make Up a Harness?

A harness is instructions, tools, environment, state, and feedback: five subsystems that sit outside the model weights. Skip one and the agent behaves erratically no matter how strong the underlying model is. Feedback, the verification loop, returns the most for the least work.

• Instructions: the project overview, hard constraints, and verification commands the agent can actually read before it starts.
• Tools: shell access, file access, tests, and the connectors it needs, scoped to least privilege without crippling it.
• Environment: pinned dependencies and a setup that runs the same way from a cold clone every time.
• State: progress and decisions written to files, because a fresh session remembers nothing that wasn't saved.
• Feedback: an executable check, lint, types, tests, a real run, that tells the agent (and you) whether the work actually holds up.

Miss any one of these and the agent looks unpredictable even though the model is fine. We wrote the full breakdown of this model in our report on harness engineering, and it's the same frame we bring to every business automation build: score each subsystem before you touch the prompt.

Does the Data Actually Back Up the Capability-Reliability Gap?

Three separate studies show the same split: AI assistance measurably speeds up a scoped task with a clear spec, and measurably slows down real work with ambiguous scope, even though developers believed the opposite was happening the whole time.

This isn't a hunch. GitHub and Accenture ran a randomized controlled trial where developers built the same HTTP server, half with Copilot, half without. The assisted group finished 55.8% faster. That's capability showing up cleanly, because the task was scoped and the definition of done was obvious: does the server respond correctly. METR ran a very different study a couple years later: 16 experienced open-source developers, working on their own mature repositories, using 2025-era AI tools including Claude. The tasks weren't scoped exercises, they were real backlog items with the usual ambiguity. Result: developers using AI took 19% longer, not shorter. Before the study, they predicted AI would save them 24% of their time. After finishing, they still believed it had saved them 20%. They were wrong both times, and the size of that gap between belief and measured outcome is the whole capability-reliability problem in one study.

The DORA number matters as much as the other two. Ninety percent of software professionals are now using AI day to day, up 14 points in a year, and DORA's own framing of the result is that AI acts as an amplifier: it makes strong engineering practices stronger and it makes weak ones worse. A harness is what determines which side of that line you land on.

How Fast Are Agents Actually Getting Better at Longer Tasks?

METR measures agent capability by the length of task an agent can finish at 50% reliability. That number has doubled roughly every seven months since 2019, and doubled every four months in 2024 and 2025 alone, an acceleration, not a plateau.

METR's time-horizon research is the closest thing this field has to a speedometer. Instead of a single benchmark score, it asks: what's the longest task, measured in how long it would take a skilled human, that a frontier model can complete half the time? That number has been doubling on a fairly steady seven-month cycle going back to 2019. In the 2024-2025 window specifically, the doubling time compressed to about four months. Models are not just getting smarter on short, scoped problems. They're getting reliably competent on longer and longer stretches of autonomous work, which is exactly where a weak harness turns from an annoyance into a total blocker, because more autonomy means more decisions made without a human catching a wrong turn in real time.

A harness does not make the model smarter. It builds a closed verification loop around the model so the capability that already exists actually gets realized.

How Do You Find Which Layer Is Actually Broken?

Run a five-layer attribution before touching the model: was the task spec written down, was the context visible, was the environment reproducible, could success be checked, was state saved. One of those five is almost always the real answer.

The diagnostic is simple and takes about twenty minutes. Pull up a failed agent run and ask, in order: was the task specified with enough detail that a right answer was even possible? Could the agent see the conventions it was supposed to follow, or were they living in someone's head? Did the environment actually match what the agent assumed, or did it burn its effort fighting a broken setup? Was there an executable way to know the work was correct, or did "looks done" get accepted as done? And did the agent have any memory of decisions made earlier in the task, or was it re-deriving everything from scratch? We cover the verification piece of this in more depth in our report on agent evaluation, because it's usually the layer that pays off fastest to fix.

What Changes Once You Build the Harness Before the Prompt?

Teams that build the harness first stop treating agent failures as a model problem and start treating them as a missing subsystem. The fix is almost never a bigger model. It's a written spec, a real environment, and a check the agent can actually run.

Once you've run the five-layer diagnostic on a handful of failed tasks, a pattern shows up fast: it's rarely the model. It's a spec that assumed shared context nobody wrote down, an environment that worked on one machine and not the next, or a definition of done that lived in someone's head instead of a file. We treat this as a design problem for every system we ship, whether it's a customer service agent handling live tickets or a back-office workflow. The two related field notes worth reading next cover the two failure modes we see most: agents that declare victory before the work actually holds up, and agents that lose the thread the moment a session ends because nothing durable was written to the repo.