Why Do Agents Declare Victory Before the Work Actually Holds Up?
Agents report done based on a feeling of completion, not a checked fact. Code that compiles and passes a shallow test looks finished, so the agent says so, even when the real behavior a user depends on was never actually run.
The pattern is always the same shape: the agent writes code, runs a narrow check, the check passes, and the agent reports success. What it skipped is everything the narrow check couldn't see, whether the feature works end to end, whether a downstream service was actually reachable, whether the change broke something that used to work. Each step from spec to code to running system loses information, and an agent under time or context pressure takes the cheapest signal available as proof. We build this exact discipline into every custom AI agent we ship, because a client-facing agent that reports success on a broken flow is worse than one that admits it's stuck.
Are AI Systems Just Bad at Judging Their Own Confidence?
Yes, and it's a known, measured property, not folklore. Modern neural networks are systematically overconfident: the confidence a model reports is reliably higher than how often it's actually correct, and the gap grows with task complexity.
This isn't specific to coding agents. Guo and colleagues showed in 2017 that modern neural networks are poorly calibrated compared to older, smaller models, meaning their stated confidence overstates their real accuracy. That finding predates today's agents by years and describes a property of the underlying systems, not a training accident anyone has since patched out. A more recent study asked a harder question directly: does an AI agent's success rate hold steady as a task gets longer, or does it decay? The answer is decay, and it isn't gentle. Success rates on long-horizon agent tasks drop off in a pattern researchers describe as an exponential half-life, meaning the odds of a clean finish fall fast as the number of steps climbs, even while the agent's own sense of how it's doing stays flat.
| Source | What it measured | Finding |
|---|---|---|
| Guo et al., "On Calibration of Modern Neural Networks" (2017) | Model confidence vs. actual accuracy across networks | Reported confidence is systematically higher than real accuracy |
| "Is There a Half-Life for the Success Rates of AI Agents?" (2025) | Agent success rate across increasing task length | Success rate decays on a roughly exponential curve as steps increase |
Do Developers Actually Notice When AI Assistance Isn't Working?
No, and a controlled study proved it. Developers using AI tools believed the tools were saving them time, both before and after finishing their tasks, while the measured result was the opposite: AI assistance made them slower.
METR's 2025 randomized controlled trial is the clearest evidence we have that this reaches past the agent to the human watching it work. Sixteen experienced open-source developers worked on real tasks in their own mature repositories, half the time with AI assistance, half without. Before starting, they forecast that AI would cut their time by 24%. After finishing, looking back at work they'd already done, they still believed AI had saved them about 20%. The measured reality: AI assistance made them 19% slower. Perception didn't correct itself even with the work sitting right in front of them. If experienced humans can't self-assess accurately after the fact, with the code and the clock both available to check, an agent grading its own output in the moment has no chance.
Think about what that gap actually implies for a workflow where a human reviews an agent's summary and moves on. The summary is written by the same system that just misjudged its own output, and the human reading it brings the same optimism bias the METR developers showed toward their own work. Two overconfident evaluations stacked on top of each other don't cancel out. They compound. That's the practical argument for never letting "the agent said it's done" stand in as your only signal, no matter how detailed or convincing the agent's own report reads.
| Measurement point | Believed effect of AI assistance | Actual measured effect |
|---|---|---|
| Forecast, before starting tasks | 24% faster | Not yet measured |
| Self-report, after completing tasks | 20% faster | 19% SLOWER |
Self-evaluation is systematically biased. The same model that generates the work flatters itself when asked to grade it. The fix is structural: separate the worker from the checker.
What Actually Stops an Agent From Self-Promoting to Done?
Four fixes work together: a done-criteria a machine can run instead of an impression, a checker that is never the same process as the worker, one task active at a time, and a tracked status file the agent cannot edit on its own authority.
The first fix is externalizing done. Every task needs a command that either passes or fails, not a description of what the agent believes it accomplished. "Tests pass" is a start; "the end-to-end flow ran against a real environment and produced the expected output" is the actual bar, because unit tests are structurally blind to the failures that only show up when components interact for real. We go deeper on why unit-level passes aren't enough in our report on AI agent evaluation.
The second fix is separating the checker from the worker. The same process that wrote the code should not be the process that decides whether the code is correct. A dedicated evaluator, even a second agent instance with a narrower, more skeptical prompt, catches issues a self-grading run waves through. Anthropic's own three-agent experiment found this directly: a bare single-agent run shipped broken core features and called itself done; the same model wrapped in a planner-generator-evaluator loop, where a separate evaluator actually clicked through the app, shipped a working one.
- Done means a passing command, never a description of intent.
- The checker runs in a separate process or prompt from the worker, and the worker cannot override its verdict.
- One task is active at a time, so a rushed agent isn't juggling five half-finished threads and reporting the whole batch done.
- Task status lives in a tracked file the agent updates only by passing verification, never by declaring itself finished.
Why Does Working One Task at a Time Reduce False Completions?
An agent working several threads at once splits its attention across all of them, and attention that's spread thin produces work that looks finished at a glance but hasn't actually been checked end to end on any single thread.
Give an agent a broad task like "add user auth" and it will touch the schema, the routes, the frontend, and quietly refactor something nearby, all in one pass, then call the whole thing done. Nothing in that pass got the attention it needed to actually verify. WIP limits fix this by drawing a hard boundary: exactly one task active, and it has to pass real verification before the next one starts. That single constraint changes what "done" means in practice, because there's no adjacent unfinished work to blur the line. It's the same discipline we apply across our own business automation builds, where a half-verified workflow change is worse than a slow one.
What Should a Feature List Look Like If an Agent Can't Self-Promote It?
A feature list works as a state machine, not a memo: each item carries a behavior, a verification command, and a state. The only way from active to passing is that command succeeding, with evidence, never the agent editing the state field itself.
A written feature list only works as a real control if the agent is structurally locked out of marking its own line item passing. That means the verification step, not the agent, owns the state transition, the same way a database constraint can't be talked out of enforcing itself. Treat the list as a primitive the rest of your process reads from, not a document someone glances at. When that's in place, an agent claiming victory stops being a trust question and becomes a simple fact you can check in one place.
What Does This Look Like on a Real Multi-Feature Build?
On a ten-feature build without a locked feature list, progress notes degrade within a few sessions into vague summaries like mostly done, and a new session has to spend real time re-inferring what actually works. A locked list with pass-state gating skips that entirely.
We've watched both versions of this play out on our own client builds. Without a structured feature list, a multi-session project's status notes drift toward something like "cart mostly done, has some bugs," which tells the next session almost nothing concrete. That session then burns real time just figuring out what "mostly" means before it can safely add anything new, and it sometimes reimplements a piece that was already working, because there was no reliable way to check. With a locked feature list where only a passing verification command can flip a state to passing, a new session reads four lines and knows exactly where things stand: which features are verified, which one is active, and which haven't started. That's the entire value of pass-state gating in one comparison, and it's why we treat a feature list as infrastructure on any agent build that spans more than a single sitting.
