Why Do Agents Declare Victory Before the Work Actually Holds Up?
Agents report done when the work looks complete to them, without checking that it is. Code that compiles and passes a shallow test looks finished, so the agent reports it as done, even when the behavior a user depends on was never actually run.
The sequence is always the same. The agent writes code, runs a narrow check, sees it pass and reports success. It skipped whatever the narrow check couldn't see, such as whether the feature works end to end, whether a downstream service was reachable, or whether the change broke something that used to work. Each step from spec to code to running system loses information, and an agent under time or context pressure takes the cheapest available signal as proof. We build checks against this into every custom AI agent we ship, because a client-facing agent that reports success on a broken flow is worse than one that says it is stuck.
Are AI Systems Just Bad at Judging Their Own Confidence?
Yes, and it is a known property that has been measured. Modern neural networks are systematically overconfident, meaning the confidence a model reports is reliably higher than how often it is actually correct, and the gap grows as tasks get longer and more complex.
The problem is not specific to coding agents. Guo and colleagues showed in 2017 that modern neural networks are poorly calibrated compared to older, smaller models, so their stated confidence overstates their real accuracy. That finding predates today's agents by years, and it describes a property of the underlying systems rather than a training accident that someone has since patched. A more recent study asked whether an AI agent's success rate holds steady as a task gets longer, and found that it decays sharply. Success rates on long-horizon agent tasks fall off in a pattern the researchers describe as an exponential half-life, so the odds of a clean finish drop fast as the number of steps climbs, even while the agent's own sense of how it is doing stays flat.
| Source | What it measured | Finding |
|---|---|---|
| Guo et al., "On Calibration of Modern Neural Networks" (2017) | Model confidence vs. actual accuracy across networks | Reported confidence is systematically higher than real accuracy |
| "Is There a Half-Life for the Success Rates of AI Agents?" (2025) | Agent success rate across increasing task length | Success rate decays on a roughly exponential curve as steps increase |
Do Developers Actually Notice When AI Assistance Isn't Working?
No. In a randomized controlled trial, experienced open-source developers using AI tools believed the tools were saving them time, both before and after finishing their tasks, while the measurements showed that AI assistance made them 19 percent slower on real tasks.
METR's 2025 randomized controlled trial is the clearest evidence we have that the problem extends past the agent to the human watching it work. Sixteen experienced open-source developers worked on real tasks in their own mature repositories, half the time with AI assistance and half without. Before starting, they forecast that AI would cut their time by 24%, and after finishing, looking back at work they had already done, they still believed AI had saved them about 20%. In fact AI assistance made them 19% slower, and their perception did not correct itself even with the finished work in front of them. If experienced people can't assess their own work accurately after the fact, with the code and the clock both available to check, an agent grading its own output in the moment has no chance of doing it.
Consider a workflow where a person reads an agent's summary and moves on. The summary was written by the same system that just misjudged its own output, and the person reading it brings the same optimism bias the METR developers showed toward their own work, so the two overconfident judgments compound instead of canceling out. This is why "the agent said it's done" should never be your only signal, however detailed or convincing the agent's own report reads.
| Measurement point | Believed effect of AI assistance | Actual measured effect |
|---|---|---|
| Forecast, before starting tasks | 24% faster | Not yet measured |
| Self-report, after completing tasks | 20% faster | 19% SLOWER |
Self-evaluation is systematically biased, because a model asked to grade work it generated rates that work too favorably. The checker has to be separate from the worker.
What Actually Stops an Agent From Self-Promoting to Done?
Four fixes used together stop it: a definition of done that a machine can run, a checker that is never the same process as the worker, one active task at a time, and a tracked status file that the agent cannot change on its own authority.
The first fix is to define done outside the agent. Every task needs a command that either passes or fails, in place of a description of what the agent believes it accomplished. "Tests pass" is a start, but the real bar is "the end-to-end flow ran against a real environment and produced the expected output," because unit tests cannot see failures that only show up when components interact for real. Our report on AI agent evaluation covers why unit-level passes aren't enough in more detail.
The second fix is to separate the checker from the worker, so the process that wrote the code is not the one that decides whether it is correct. A dedicated evaluator, even a second agent instance with a narrower and more skeptical prompt, catches issues that a self-grading run lets through. Anthropic's own three-agent experiment showed this. A bare single-agent run shipped broken core features and reported itself done, while the same model wrapped in a planner-generator-evaluator loop, where a separate evaluator clicked through the app, shipped a working one.
- Done means a passing command, never a description of intent.
- The checker runs in a separate process or prompt from the worker, and the worker cannot override its verdict.
- One task is active at a time, so a rushed agent isn't juggling five half-finished threads and reporting the whole batch done.
- Task status lives in a tracked file the agent updates only by passing verification, never by declaring itself finished.
Why Does Working One Task at a Time Reduce False Completions?
An agent working several threads at once splits its attention across all of them, which produces work that looks finished at a glance but has not been checked end to end on any single thread. Limiting the agent to one active task means each piece is verified before the next starts.
Give an agent a broad task like "add user auth" and it will touch the schema, the routes and the frontend, refactor something nearby while it is there, and then call the whole thing done, without any part of that pass getting enough attention to be verified. A WIP limit sets a hard boundary of exactly one active task, which has to pass real verification before the next one starts. That one constraint changes what "done" means in practice, because no unfinished work sits alongside the task to blur the line. We apply the same rule to our own business automation builds, where a half-verified workflow change is worse than a slow one.
What Should a Feature List Look Like If an Agent Can't Self-Promote It?
The feature list should work as a state machine, where each item carries a behavior, a verification command and a state. An item moves from active to passing only when that command succeeds with evidence, and the agent never edits the state field itself.
A written feature list only works as a control if the agent is locked out of marking its own line item passing. The verification step owns the state transition, in the same way a database constraint enforces itself no matter what the caller says. The rest of your process should read status from the list instead of treating it as a document someone glances at. Once that is in place, whether the agent finished is a fact you can check in one place, and you no longer have to decide whether to trust its claim.
What Does This Look Like on a Real Multi-Feature Build?
On a ten-feature build without a locked feature list, progress notes degrade within a few sessions into vague summaries like mostly done, and a new session has to spend real time working out what actually works. With a locked list and pass-state gating, a new session reads the current state directly.
We have seen both versions on our own client builds. Without a structured feature list, a multi-session project's status notes drift toward something like "cart mostly done, has some bugs," which tells the next session almost nothing concrete. That session then spends real time working out what "mostly" means before it can safely add anything, and it sometimes reimplements a piece that was already working because there was no reliable way to check. With a locked feature list, where only a passing verification command can flip a state to passing, a new session reads four lines and knows which features are verified, which one is active and which haven't started. That difference is why we treat a feature list as infrastructure on any agent build that spans more than a single sitting.
