What Does Over-Engineering Look Like in an Agent Harness?
In our harness it looked like 63,535 lines of hook code, of which only 12,713 belonged to the 20 wired hooks that act on an agent. About 80 percent was apparatus for checking the harness itself. Of 25 wired scripts, 6 prevent a mistake, and the rest observe, log or re-check other checkers.
A harness is the tooling around an AI agent, such as the gates that block a bad commit, the hooks that inject context and the logs that record what happened. Ours had grown to 63,535 lines under its hooks directory. When we separated the code that acts on an agent from the code that checks the harness itself, the 20 wired top-level hooks accounted for 12,713 lines. The other 80 percent was checkers that checked other checkers, fixtures for those checkers, and reporters on the health of both.
Of 25 wired scripts, only 6 prevent a mistake, and the rest observe and report. That reporting cost tokens. When we ran all 12 SessionStart hooks and captured the output, the status banner injected into every session before any work begins measured 1,570 tokens (6,297 bytes). We counted roughly 439 main-session transcripts in one 24 hour window, which works out to about 700,000 tokens per day spent describing the harness to itself. The slowest single startup check took 47.0 seconds, timed twice.
Why Did Adding LLM Observability Produce 120 True Issues?
An audit of our own tooling kept producing real findings about that tooling for as long as we kept looking, and all 120 were true. On two ordinary days with no audit, agents worked 10 and 8 hours and filed zero issues. On three audit days they filed 1.50, 0.75 and 3.25 per active hour, with no downward trend.
The audit was meant to be the first step of an LLM observability build, on the reasoning that we should understand the instrument before trusting its traces. Counted in UTC days, it filed 24 issues on day one, 18 on day two and 78 on day three, while closures ran 4, 3 and 51. At the peak, 81 issues were open. Three independent deduplication passes found only 9 issues covering 6 shared defects, and the rest were distinct. Every issue we checked described a real problem.
The evidence came from the arrival process rather than from the content of the issues. Over 65 fixed local-hour buckets, we first reported a correlation of r = 0.826 between issue arrivals and agent-active hours. An independent verification pass could not reproduce it under any of four definitions and rejected it for three reasons: the agent is the one filing the issues, so the correlation is definitional; controlling for hour of day drops it to p = 0.085; and it could never have separated the two hypotheses anyway. The conclusion held up on better evidence from a natural experiment. All 117 bucketed arrivals landed in hours when an agent was active, and the 21 inactive buckets produced zero. Arrivals ran 2.79 per closure across the whole period. That is the pattern of work bound by how much attention is dispatched rather than by how many defects exist. Defect-bound work slows as the pool of defects drains, while ours scaled with headcount and stopped as soon as we stopped looking.
We never examined 69.6 percent of the 115 tracked script files under hooks, so the audit did not end by exhausting the surface. Of those 115 files, 49 (42.6 percent) were added during the audit itself, which means the audit grew the code it was auditing. The defects were still real: in a sample of every tenth issue, 10 of 11 subjects predated the audit.
Every one of the 120 issues was true. What went wrong was that true findings have no natural limit, and nobody had set a budget for them.
How Wrong Were Our Own Numbers?
We caught four of our own numbers wrong, each through a cheap control. A grep for API keys matched the word risk and returned 251 phantom hits, and a failure count was double its true value by construction. An overnight control was false, and our estimate of two broken sessions was an unknown we could not enumerate.
These are the confident numbers we got wrong during the same three days, with what caught each one. We are including them because a post that reported only the flattering measurements would not be worth publishing.
- "251 files contain an API key." The predicate matched the word risk, as in risk-weighted, and the real count in the target tree was zero.
- "48 job failures, 46 silent." The grep counted both the raised error and the source line echoed in every traceback, which doubles the count by construction. The real answer was 26 and 24.
- "Arrivals go to zero overnight, so nobody was looking." This control was false, because a monitoring session accounted for 7 active agent-hours overnight, so zero arrivals proved nothing.
- "The harness caused two broken sessions." The defensible floor is 7, and the upper bound, counting every issue that names a harness artifact as the agent of harm, is 15.
- "45 copies of one script across our code directory, 6 diverged versions." We measured 71 worktrees of one repository, 6 hashes that were 6 commits of a single history, and zero cross-repo drift.
- A test suite that isolated its reads but not its writes had been writing dated fixtures into the live off-machine backup directory of a production database, then running the retention prune over them. The fixtures pruned themselves, which is why nobody saw it.
In all six cases the first number came from a plausible one-liner, and the correction came from a control that took minutes to run. Re-deriving a count before repeating it is cheaper than publishing it wrong.
What Did the AI Agent Observability Data Actually Say?
It said mostly that the telemetry was watching itself. Of 2,966 health-log rows in 24 hours, 1,734 (58 percent) were reporters re-logging standing conditions about the reporting system. Across 542 decoded gate decisions, none was a block on real work, so the observability we had built never changed a decision.
AI agent observability is supposed to tell you whether the agent did what it claimed, and whether knowing what it did would change any decision you make. Our telemetry mostly reported whether the telemetry was still running. In one 24 hour window the health log recorded 2,966 rows, and 1,734 of them (58 percent) were reporters re-logging standing conditions about the reporting apparatus itself.
The decision channels had even less in them. The two gate logs we could decode held 542 rows, and none of them recorded a block or a deny. In one gate, only 2 of 166 rows ever reached a subject at all, and both subjects were synthetic test inputs, so the number of live decisions that gate changed is zero. That one measurement captures the LLM ops failure mode, where a pipeline observes everything and never affects a decision.
What Does a Sane LLM Ops Budget Keep?
A sane budget keeps the parts that block mistakes and fixes the best of them, and six blocking gates survived our cut. The component that catches real errors fires at 23 percent measured precision, against the 85 percent its own header claimed, and one added condition takes it to roughly 75 to 80 percent.
We did not end the audit by deleting everything. Six blocking gates survived, because blocking a mistake is the one job that pays for its own overhead. The most instructive survivor is the hook that checks whether a command actually ran before an agent claims it did, which fired 821 times in the retained live log. Its header claimed 85 percent precision, but that figure measured whether a status was masked and said nothing about whether the verdict would change anything.
Two model seats judged the same 40-row sample independently and landed two points apart, finding that 23.1 percent and 25.0 percent of firings would actually change a decision. The split by arm told us more than the aggregate. The arm that catches work which never ran scores 60 percent and should be left alone. The arm that flags a masked exit status scores 20 percent, and it was firing on agents who had just confirmed a push against the server, which is exactly the habit the harness exists to teach. The fix is one added condition. In the sample, 4 of 9 true catches occurred where the exit status immediately follows the pipeline, and 0 of 24 harmless rows ever consumed the status at all. Suppressing firings where the status is never consumed keeps 8 of 9 true catches and projects to 75 to 80 percent precision.
We would rather budget audits than stop doing them. In practice that means setting an issue budget before an audit starts, capping arrivals per closure, and requiring every observer to name the decision it can change, deleting the ones that cannot. Our audit found nothing false, but it had no number that told it to stop.

