Over-Engineering, Measured: 120 True Issues in Three Days

What Does Over-Engineering Look Like in an Agent Harness?

In our harness it looked like 63,535 lines of hook code where only 12,713 belonged to the 20 wired hooks that act on an agent. About 80 percent was apparatus checking the harness itself. Of 25 wired scripts, 6 prevent a mistake. The rest observe, log, or re-check other checkers.

A harness is the tooling around an AI agent: the gates that block a bad commit, the hooks that inject context, the logs that record what happened. Ours had grown to 63,535 lines under its hooks directory. When we separated the code that acts on an agent from the code that checks the harness itself, the 20 wired top-level hooks accounted for 12,713 lines. The other 80 percent was apparatus: checkers checking checkers, fixtures for the checkers, and reporters on the health of both.

Of 25 wired scripts, only 6 actually prevent a mistake. The rest observe and report. The reporting was not free: the status banner injected into every session before any work begins measured 1,570 tokens (6,297 bytes) when we ran all 12 SessionStart hooks and captured the output. Across the roughly 439 main-session transcripts we counted in one 24 hour window, that is about 700,000 tokens per day spent announcing the harness to itself. The slowest single startup check took 47.0 seconds, timed twice.

Why Did Adding LLM Observability Produce 120 True Issues?

Because auditing an instrument produces instrument findings without bound. All 120 issues were true; none were false positives. Arrivals tracked the sentence that funded them, not the defect supply. On two ordinary working days with no audit running, agents worked 10 and 8 hours and filed zero issues; on the three audit days they filed 1.50, 0.75 and 3.25 per active hour with no downward trend. The work was dispatch-bound, not defect-bound, and stopped when we stopped looking.

The audit was supposed to be the first step of an LLM observability build: know the instrument before you trust its traces. It filed 24 issues on day one, 36 on day two, 57 on day three. Closures ran 4, 8, and 31. At peak, 81 issues were open. Three independent deduplication passes found only 9 issues covering 6 shared defects; the rest were distinct. Every issue we checked described a real problem.

The tell was the arrival process, not the content. Over 65 fixed local-hour buckets, we first reported a correlation of r = 0.826 between issue arrivals and agent-active hours. An independent verification pass could not reproduce it under any of four definitions, and killed it three ways: the agent is the one filing the issues, so the correlation is definitional; controlling for hour of day drops it to p = 0.085; and it could never have separated the two hypotheses anyway. The conclusion survived on better evidence, which is a natural experiment rather than a correlation. All 117 bucketed arrivals landed in hours when an agent was active; the 21 inactive buckets produced zero. Arrivals ran 2.79 per closure across the whole period. That is the signature of work bound by how much attention you dispatch, not by how many defects exist. Defect-bound work slows as the defect pool drains. Ours scaled with headcount and stopped the moment we stopped looking.

Two more numbers complete the picture. We never examined 69.6 percent of the 115 tracked script files under hooks, so the audit did not end by exhausting the surface. And 49 of those 115 files, 42.6 percent, were added during the audit itself: the audit grew the thing it was auditing. The defects were still real. In a sample of every tenth issue, 10 of 11 subjects predated the audit.

Every one of the 120 issues was true. The failure was never false positives. It was that truth is unbounded and nobody set a budget.

How Wrong Were Our Own Numbers?

Wrong four times that we caught, each by a cheap control. A grep for API keys matched the word risk and returned 251 phantom hits. A failure count was double its true value by construction. An overnight control was false. Our harm estimate of two broken sessions was an unknown number we could not enumerate afterward, which is its own finding.

A post that only reports the flattering measurements is not worth publishing, so here is the ledger of confident numbers we got wrong during the same three days, and what caught each one.

• "251 files contain an API key." The predicate matched the word risk, as in risk-weighted. Real answer in the target tree: zero.
• "48 job failures, 46 silent." The grep counted both the raised error and the source line echoed in every traceback. Real answer: 26 and 24. Off by exactly 2x, by construction.
• "Arrivals go to zero overnight, so nobody was looking." A false control: a monitoring session put 7 active agent-hours overnight, so zero arrivals proved nothing.
• "The harness caused two broken sessions." The defensible floor is 7. The upper bound, counting every issue that names a harness artifact as the agent of harm, is 15.
• "45 copies of one script across our code directory, 6 diverged versions." Measured: 71 worktrees of one repository, 6 hashes that were 6 commits of a single history, zero cross-repo drift.
• A test suite that isolated its reads but not its writes had been writing dated fixtures into the live off-machine backup directory of a production database, then running the retention prune over them. The fixtures pruned themselves, which is why nobody saw it.

The pattern across all six: the first number was produced by a plausible one-liner, and the correction came from a control that cost minutes. Re-deriving a count before repeating it is cheaper than publishing it wrong.

What Did the AI Agent Observability Data Actually Say?

Mostly that the telemetry watched itself. Of 2,966 health-log rows in 24 hours, 1,734, or 58 percent, were reporters re-logging standing conditions about the reporting system. Across 542 decoded gate decisions, zero were blocks on real work. Observability that never changes a decision is inventory, not insight.

AI agent observability is supposed to answer two questions: did the agent do what it claimed, and would you decide anything differently knowing what it did. Our telemetry mostly answered a third question nobody asked: is the telemetry still there. In one 24 hour window the health log recorded 2,966 rows, and 1,734 of them, 58 percent, were reporters re-logging standing conditions about the reporting apparatus itself.

The decision channels were emptier still. Across the two gate logs we could decode, 542 rows, the number of recorded blocks or denies was zero. Only 2 of 166 rows in one gate ever reached a subject at all, and both subjects were synthetic test inputs. The live count of decisions changed by that gate is zero. That is the LLM ops failure mode in one measurement: a pipeline that observes everything and decides nothing.

What Does a Sane LLM Ops Budget Keep?

Keep the parts that block mistakes and fix the best of them. Six blocking gates survived our cut. The one component that catches real errors fires at 23 percent measured precision, against the 85 percent its own header claimed, and one added condition takes it to roughly 75 to 80.

The end of the audit was not deletion of everything. Six blocking gates survived, because blocking a mistake is the one job that pays for its own overhead. The most instructive survivor is the hook that checks whether a command actually ran before an agent claims it did. It fired 821 times in the retained live log. Its own header claimed 85 percent precision, but that figure measured whether a status was masked, not whether the verdict would change anything.

Two model seats judged the same 40-row sample independently and landed two points apart: 23.1 percent and 25.0 percent of firings would actually change a decision. Splitting by arm mattered more than the aggregate did. The arm that catches work which never ran scores 60 percent and should be left alone. The arm that flags a masked exit status scores 20 percent, and it was firing on agents who had just confirmed a push against the server, which is the exact habit the harness exists to teach. The fix is one condition. In the sample, 4 of 9 true catches occurred where the exit status immediately follows the pipeline, and 0 of 24 harmless rows ever consumed the status at all. Suppressing firings where the status is never consumed keeps 8 of 9 true catches and projects to 75 to 80 percent precision.

That is the lesson we are keeping: a budget, not abstinence. Set an issue budget before an audit starts. Cap arrivals per closure. Require every observer to name the decision it can change, and delete the ones that cannot. The audit did not fail by finding false things. It failed by having no number that told it to stop.