Over-Engineering, Measured: 120 True Issues in Three Days

We set out to add an LLM ops component to our agent harness: traces, health telemetry, gates. Before building it, we audited what was already there. Three days later we had filed 120 issues against our own tooling and the audit was still finding more.

Category
Report
Published
Updated

This report covers what those three days measured, including the numbers that make us look bad, which were the most useful.

Short answer

We audited our own AI agent harness for three days and filed 120 issues against it. Every issue was true, and the trouble was that checking an instrument yields true findings for as long as someone looks. The arrival rate tracked what we funded rather than what was broken, so the work was dispatch-bound, not defect-bound. Our remedy is an issue budget set before an audit starts.

Tree · 8 min read

What Does Over-Engineering Look Like in an Agent Harness?

In our harness it looked like 63,535 lines of hook code, of which only 12,713 belonged to the 20 wired hooks that act on an agent. About 80 percent was apparatus for checking the harness itself. Of 25 wired scripts, 6 prevent a mistake, and the rest observe, log or re-check other checkers.

A harness is the tooling around an AI agent, such as the gates that block a bad commit, the hooks that inject context and the logs that record what happened. Ours had grown to 63,535 lines under its hooks directory. When we separated the code that acts on an agent from the code that checks the harness itself, the 20 wired top-level hooks accounted for 12,713 lines. The other 80 percent was checkers that checked other checkers, fixtures for those checkers, and reporters on the health of both.

FIG AWhere the hook code actually went

Of 25 wired scripts, only 6 prevent a mistake, and the rest observe and report. That reporting cost tokens. When we ran all 12 SessionStart hooks and captured the output, the status banner injected into every session before any work begins measured 1,570 tokens (6,297 bytes). We counted roughly 439 main-session transcripts in one 24 hour window, which works out to about 700,000 tokens per day spent describing the harness to itself. The slowest single startup check took 47.0 seconds, timed twice.

Why Did Adding LLM Observability Produce 120 True Issues?

An audit of our own tooling kept producing real findings about that tooling for as long as we kept looking, and all 120 were true. On two ordinary days with no audit, agents worked 10 and 8 hours and filed zero issues. On three audit days they filed 1.50, 0.75 and 3.25 per active hour, with no downward trend.

The audit was meant to be the first step of an LLM observability build, on the reasoning that we should understand the instrument before trusting its traces. Counted in UTC days, it filed 24 issues on day one, 18 on day two and 78 on day three, while closures ran 4, 3 and 51. At the peak, 81 issues were open. Three independent deduplication passes found only 9 issues covering 6 shared defects, and the rest were distinct. Every issue we checked described a real problem.

FIG BIssue arrivals vs closures per day (UTC)

The evidence came from the arrival process rather than from the content of the issues. Over 65 fixed local-hour buckets, we first reported a correlation of r = 0.826 between issue arrivals and agent-active hours. An independent verification pass could not reproduce it under any of four definitions and rejected it for three reasons: the agent is the one filing the issues, so the correlation is definitional; controlling for hour of day drops it to p = 0.085; and it could never have separated the two hypotheses anyway. The conclusion held up on better evidence from a natural experiment. All 117 bucketed arrivals landed in hours when an agent was active, and the 21 inactive buckets produced zero. Arrivals ran 2.79 per closure across the whole period. That is the pattern of work bound by how much attention is dispatched rather than by how many defects exist. Defect-bound work slows as the pool of defects drains, while ours scaled with headcount and stopped as soon as we stopped looking.

We never examined 69.6 percent of the 115 tracked script files under hooks, so the audit did not end by exhausting the surface. Of those 115 files, 49 (42.6 percent) were added during the audit itself, which means the audit grew the code it was auditing. The defects were still real: in a sample of every tenth issue, 10 of 11 subjects predated the audit.

Every one of the 120 issues was true. What went wrong was that true findings have no natural limit, and nobody had set a budget for them.

How Wrong Were Our Own Numbers?

We caught four of our own numbers wrong, each through a cheap control. A grep for API keys matched the word risk and returned 251 phantom hits, and a failure count was double its true value by construction. An overnight control was false, and our estimate of two broken sessions was an unknown we could not enumerate.

These are the confident numbers we got wrong during the same three days, with what caught each one. We are including them because a post that reported only the flattering measurements would not be worth publishing.

  • "251 files contain an API key." The predicate matched the word risk, as in risk-weighted, and the real count in the target tree was zero.
  • "48 job failures, 46 silent." The grep counted both the raised error and the source line echoed in every traceback, which doubles the count by construction. The real answer was 26 and 24.
  • "Arrivals go to zero overnight, so nobody was looking." This control was false, because a monitoring session accounted for 7 active agent-hours overnight, so zero arrivals proved nothing.
  • "The harness caused two broken sessions." The defensible floor is 7, and the upper bound, counting every issue that names a harness artifact as the agent of harm, is 15.
  • "45 copies of one script across our code directory, 6 diverged versions." We measured 71 worktrees of one repository, 6 hashes that were 6 commits of a single history, and zero cross-repo drift.
  • A test suite that isolated its reads but not its writes had been writing dated fixtures into the live off-machine backup directory of a production database, then running the retention prune over them. The fixtures pruned themselves, which is why nobody saw it.

In all six cases the first number came from a plausible one-liner, and the correction came from a control that took minutes to run. Re-deriving a count before repeating it is cheaper than publishing it wrong.

What Did the AI Agent Observability Data Actually Say?

It said mostly that the telemetry was watching itself. Of 2,966 health-log rows in 24 hours, 1,734 (58 percent) were reporters re-logging standing conditions about the reporting system. Across 542 decoded gate decisions, none was a block on real work, so the observability we had built never changed a decision.

AI agent observability is supposed to tell you whether the agent did what it claimed, and whether knowing what it did would change any decision you make. Our telemetry mostly reported whether the telemetry was still running. In one 24 hour window the health log recorded 2,966 rows, and 1,734 of them (58 percent) were reporters re-logging standing conditions about the reporting apparatus itself.

FIG CWhat the health telemetry was about

The decision channels had even less in them. The two gate logs we could decode held 542 rows, and none of them recorded a block or a deny. In one gate, only 2 of 166 rows ever reached a subject at all, and both subjects were synthetic test inputs, so the number of live decisions that gate changed is zero. That one measurement captures the LLM ops failure mode, where a pipeline observes everything and never affects a decision.

What Does a Sane LLM Ops Budget Keep?

A sane budget keeps the parts that block mistakes and fixes the best of them, and six blocking gates survived our cut. The component that catches real errors fires at 23 percent measured precision, against the 85 percent its own header claimed, and one added condition takes it to roughly 75 to 80 percent.

We did not end the audit by deleting everything. Six blocking gates survived, because blocking a mistake is the one job that pays for its own overhead. The most instructive survivor is the hook that checks whether a command actually ran before an agent claims it did, which fired 821 times in the retained live log. Its header claimed 85 percent precision, but that figure measured whether a status was masked and said nothing about whether the verdict would change anything.

FIG Dcommand-ran-check precision: claimed, measured, and by arm

Two model seats judged the same 40-row sample independently and landed two points apart, finding that 23.1 percent and 25.0 percent of firings would actually change a decision. The split by arm told us more than the aggregate. The arm that catches work which never ran scores 60 percent and should be left alone. The arm that flags a masked exit status scores 20 percent, and it was firing on agents who had just confirmed a push against the server, which is exactly the habit the harness exists to teach. The fix is one added condition. In the sample, 4 of 9 true catches occurred where the exit status immediately follows the pipeline, and 0 of 24 harmless rows ever consumed the status at all. Suppressing firings where the status is never consumed keeps 8 of 9 true catches and projects to 75 to 80 percent precision.

We would rather budget audits than stop doing them. In practice that means setting an issue budget before an audit starts, capping arrivals per closure, and requiring every observer to name the decision it can change, deleting the ones that cannot. Our audit found nothing false, but it had no number that told it to stop.

Common Questions

What is over-engineering in software?
Over-engineering is building more mechanism than the problem needs, such as extra checks, layers or generality. It is hard to spot because each added piece is justified locally. In our case every piece was true and useful in isolation, and the excess only showed up in aggregate measurements, such as the 80 percent of hook code that existed to check the harness.
How do you tell over-engineering from diligence?
Measure the arrival process. Defect-bound work slows as the defect pool drains, while dispatch-bound work scales with attention and stops when you stop looking. Ours did: on two ordinary days with no audit, agents worked 10 and 8 hours and filed zero issues.
Were any of the 120 issues false positives?
No. Every issue we checked described a real problem, and in a sample, 10 of 11 subjects predated the audit. That made the volume risky, because review filters out false positives while true findings keep passing it, so nothing limits them unless someone sets a budget.
What is LLM observability?
LLM observability is the practice of recording what a model-driven system actually did, including traces of calls, tool use, decisions, costs and failures. It is necessary but not sufficient, since observing tells you whether the system is healthy and only evaluation tells you whether the output is good. Our telemetry recorded a lot and changed no decisions.
How much observability does an AI agent harness need?
Enough to tell you whether the thing actually ran, and whether the answer would change any decision you make. We would keep gates that block mistakes and telemetry that feeds a named decision, and give every observer a budget. We kept 6 blocking gates out of 25 wired scripts, and we are fixing one of them instead of adding more.
Why publish the numbers that make you look bad?
They show that the method works. Four of our own confident numbers were wrong, and each correction came from a cheap control anyone can run, such as re-deriving the count, checking what the grep actually matched or testing the control condition. We do not think a report that includes only the flattering measurements counts as measurement.

Sources

SourcePublisherLink
Building Effective AI AgentsAnthropicwww.anthropic.com
Langfuse Documentation: Open Source LLM ObservabilityLangfuselangfuse.com
LangSmith ObservabilityLangChaindocs.langchain.com
A Philosophy of Software DesignJohn Ousterhout, Stanford Universityweb.stanford.edu
Goodhart's LawWikipediaen.wikipedia.org

Questions to explore next

  • What is the right issue budget for an audit of a system this size?
  • Does arrivals-per-closure predict over-engineering earlier than the correlation test?
  • How does the fixed command-ran-check perform after a month of live firings?

Keep Exploring

Chalk stick figure in a hard hat presenting a little machine of blue gears it just built

Find the leverage
already in your business.

Building What's Next.