Why AI Agents Fail on Tasks They Can Actually Do

Why Does a Capable Model Still Fail at Real Work?

A capable model still fails at real work because raw capability and an agent's reliability are different things. The same model that fails a task with a vague brief, no real environment, and no way to check its own work finishes that exact task once those three things exist, so the gap is rarely intelligence.

Capability is what a model can do in principle, and reliability is whether it does that thing correctly, on your data and in your environment, without a person watching over its shoulder. Benchmarks measure capability, production measures reliability, and the two numbers are not close. The strongest public coding agents resolve roughly half of hand-picked SWE-bench Verified issues, which already come with clean descriptions and a ready-made test. Give them the kind of vague ticket a real engineer gets, with no tests and undocumented conventions, and the success rate drops hard. The model has not gotten worse. What is missing is everything that used to compensate for the ambiguity, such as a coworker's context and a shared understanding of the codebase.

We build custom AI agents for a living, and we have seen the same experiment play out on our own work more times than we'd like to admit: the same model and the same prompt produce wildly different outcomes because one run had a written definition of done and the other didn't. Anthropic ran a version of this test formally, with one prompt and one model, comparing a bare single-agent run against a run wrapped in a planner, generator, and evaluator loop. The bare run finished in 20 minutes for $9 and shipped a broken game. The harnessed run took six hours and $200 and shipped a fully playable one, with no change to the model.

What Are the Five Subsystems That Make Up a Harness?

The five subsystems of a harness are instructions, tools, environment, state, and feedback, and all of them sit outside the model weights. Skip one and the agent behaves erratically no matter how strong the underlying model is. Feedback, the verification loop, returns the most for the least work.

• Instructions: the project overview, hard constraints, and verification commands the agent can actually read before it starts.
• Tools: shell access, file access, tests, and the connectors it needs, scoped to least privilege without crippling it.
• Environment: pinned dependencies and a setup that runs the same way from a cold clone every time.
• State: progress and decisions written to files, because a fresh session remembers nothing that wasn't saved.
• Feedback: an executable check (lint, types, tests, a real run) that tells the agent, and you, whether the work actually holds up.

Miss any one of these and the agent looks unpredictable even though the model is fine. We wrote the full breakdown of this model in our report on harness engineering, and it's the same frame we bring to every business automation build: score each subsystem before you touch the prompt.

Does the Data Actually Back Up the Capability-Reliability Gap?

Yes, three separate studies show the same split. AI assistance measurably speeds up a scoped task with a clear spec and measurably slows down real work with ambiguous scope in mature repositories, even though developers believed the opposite was happening the whole time.

GitHub and Accenture ran a randomized controlled trial where developers built the same HTTP server, half with Copilot and half without, and the assisted group finished 55.8% faster. That is capability showing up cleanly, because the task was scoped and the definition of done was obvious: whether the server responds correctly. METR ran a very different study a couple of years later with 16 experienced open-source developers working on their own mature repositories, using 2025-era AI tools including Claude. Their tasks were real backlog items with the usual ambiguity rather than scoped exercises, and developers using AI took 19% longer. Before the study, they predicted AI would save them 24% of their time, and after finishing they still believed it had saved them 20%. They were wrong both times, and that gap between belief and measured outcome is the capability-reliability problem showing up inside a single study.

The DORA number matters as much as the other two. Ninety percent of software professionals now use AI day to day, up 14 points in a year, and DORA frames the result as AI acting as an amplifier that makes strong engineering practices stronger and weak ones worse. A harness is what decides which side of that line a team lands on.

How Fast Are Agents Actually Getting Better at Longer Tasks?

The length of task an agent can finish at 50% reliability, which is how METR measures agent capability, has doubled roughly every seven months since 2019. In 2024 and 2025 alone it doubled every four months, so the pace is accelerating.

METR's time-horizon research is the closest thing this field has to a speedometer. Instead of a single benchmark score, it measures the longest task, sized by how long it would take a skilled human, that a frontier model can complete half the time. That number has been doubling on a fairly steady seven-month cycle going back to 2019, and in the 2024-2025 window the doubling time compressed to about four months. Beyond short, scoped problems, models are becoming reliably competent on longer and longer stretches of autonomous work. That is where a weak harness turns from an annoyance into a total blocker, because more autonomy means more decisions made without a human catching a wrong turn in real time.

A harness leaves the model as smart as it was and builds a closed verification loop around it, so the capability the model already has actually gets realized.

How Do You Find Which Layer Is Actually Broken?

Find the broken layer with a five-layer attribution before touching the model: was the task spec written down, was the context visible, was the environment reproducible, could success be checked, and was state saved between sessions? One of those five layers is almost always where the failure is.

The diagnostic is simple and takes about twenty minutes. Pull up a failed agent run and ask, in order: was the task specified with enough detail that a right answer was even possible? Could the agent see the conventions it was supposed to follow, or were they living in someone's head? Did the environment actually match what the agent assumed, or did it burn its effort fighting a broken setup? Was there an executable way to know the work was correct, or did "looks done" get accepted as done? And did the agent have any memory of decisions made earlier in the task, or was it re-deriving everything from scratch? We cover the verification piece of this in more depth in our report on agent evaluation, because it's usually the layer that pays off fastest to fix.

What Changes Once You Build the Harness Before the Prompt?

Teams that build the harness first stop treating agent failures as a model problem and start looking for the missing subsystem. The fixes they end up making are a written spec, a real environment, and a check the agent can actually run, and almost never a bigger model.

Run the five-layer diagnostic on a handful of failed tasks and a pattern shows up fast. The cause is rarely the model. More often it is a spec that assumed shared context nobody wrote down, an environment that worked on one machine and not the next, or a definition of done that lived in someone's head instead of a file. We treat this as a design problem for every system we ship, whether it's a customer service agent handling live tickets or a back-office workflow. The two related field notes worth reading next cover the two failure modes we see most: agents that declare the work finished before it holds up, and agents that lose the thread the moment a session ends because nothing durable was written to the repo.