How Does the Overnight Pipeline Work?
GitHub issues run in overnight waves through a sandboxed loop. An LLM agent works each issue serially on a wave branch inside Docker and git worktree isolation. A bash script judges the result: ruff lint against a baseline, ruff format on changed files, the full pytest suite, and a green GitHub Actions run on the exact branch tip.
One rule anchors the design: the checker is never the worker. The worker is an LLM, so the judge is a bash script on purpose. It runs ruff lint against a recorded baseline, ruff format on the files the agent changed, the full pytest suite, and then requires a green GitHub Actions run on the exact commit at the branch tip. Local green and CI green, same sha, or the issue stays open.
A script this dumb cannot be argued with because it cannot reason. An LLM judge can be talked into a pass by an LLM worker, or can talk itself into one. Exit codes have no such weakness. When the judge says red, something is red, and the only question left is what.
What Happened When Wave 10 Came Back Red?
Wave 10 carried three issues, two test-isolation fixes and one dashboard feature. All three came back red. The obvious reading was that the agents had written bad code overnight. The logs said otherwise: the failures sat in files no agent in the wave had touched.
Wave 10 was small: two test-isolation fixes and one dashboard feature. A wave that ordinary going fully red usually means one of two things, either the agents had a bad night or something under them moved. Our first instinct was the agents. Most people's would be.
The verifier log pointed somewhere else. The format gate had failed on 18 files, and none of the three issues in the wave had touched any of them.
Why Did 18 Files the Agents Never Edited Fail Formatting?
The format gate flagged 18 files the wave never edited. The gate had been broken by a bash 3.2 portability bug on macOS and had been passing without checking anything. The previous wave fixed it, so its first working run surfaced weeks of formatting debt that had built up on trunk.
The format gate in our verifier had been broken for weeks. The script used bash syntax that macOS's ancient bash 3.2 does not support, so on our hardware the gate exited green without checking a single file. We missed it for weeks, because a gate that always passes looks identical to a codebase that is always clean.
The previous wave had fixed that portability bug as part of unrelated hardening. Wave 10 was the first wave judged by a format gate that worked. Its first working run found every file that had drifted while the gate was asleep: 18 of them, none touched by the agents on trial.
The debt comes due the day the gate starts working.
We paid it down, ran the verifier again, and got our second surprise. Still red.
Why Were Two Tests Out of 3,354 Still Failing?
Two tests out of 3,354 failed after the formatting debt was paid. The fixture used absolute dates, July 11 through 13, and the assertion checked them against a rolling 100-hour freshness window. The tests passed the day before and failed the next because the calendar moved. A helper that re-stamps fixture dates relative to now fixed them for good.
The failing tests checked a freshness rule: is this data recent, where recent means inside a rolling 100-hour window? The fixture feeding them used absolute dates, July 11 through July 13, written in as literals. On the day the tests were written, those dates sat comfortably inside the window. Then the window kept rolling and the dates stayed put.
So the tests passed on the day they were merged, passed the next day, and failed the day after that with no code change anywhere. An agent had written that fixture weeks earlier. It was a time bomb with a calendar for a fuse, and it happened to go off during wave 10. The fix was a small helper that re-stamps fixture dates relative to the current time, so the data is always the same age no matter when the suite runs.
Agent-written tests have a taste for concrete example data, and concrete dates rot. If your agents write tests, grep the fixtures for date literals before the calendar does it for you.
So Were the Agents Ever Guilty?
No. The agent-written changes in wave 10 were fine as submitted. One red came from a gate that had started working, the other from a stale fixture already on trunk. Both looked like agent failures at first glance, which is the argument for letting a script, not a hunch, decide.
Once the debt was paid and the fixture re-stamped, wave 10's work passed every gate as written. The agents had done their jobs the whole time. Every red that night came from the harness getting stricter or from latent debt on trunk choosing that moment to surface.
Where wave 10's red actually came from
Source: Wave 10 verifier logs, 2026-07-15| Red | Looked like | Turned out to be | The fix |
|---|---|---|---|
| First verifier run | Agents shipped unformatted code | 18 files of trunk debt, exposed by the format gate's first working run | Format the backlog, keep the gate |
| Second verifier run | Agents broke two tests | Absolute fixture dates aged past a rolling 100-hour freshness window | Re-stamp fixture dates relative to now |
Twice in one night we read "red wave" as "bad agents," and twice the evidence pointed at our own infrastructure. That bias is worth naming because it decides where you spend your debugging hours. We wrote up the general version of this rule in our report on harness engineering: when an agent fails, audit the harness before you blame the model. Wave 10 is what that rule looks like at 7 a.m. with the logs open.
What Should You Steal From This?
Six rules survived the night: fix the harness before you blame the model, keep the judge too dumb to persuade, demand green locally and in CI on the same commit, expect new gates to surface old debt, audit agent-written tests for hard-coded dates, and never let a worker close its own issue.
- Fix the harness, not the model. Both failures lived in our infrastructure. Neither would have been solved by a smarter agent, a better prompt, or a bigger model.
- Keep the judge dumb. A bash script cannot be persuaded, and that is the point. Our verifier being incorruptible is why we trusted the red enough to keep digging instead of overriding it.
- Require two green signals. Local checks and a CI run on the exact branch tip. The bash 3.2 bug is the case for this: our local gate lied for weeks while claiming to work.
- Expect a new gate to eat old debt on day one. The first honest run of any checker will bill you for every day it was broken. Budget for that instead of reading it as a regression.
- Audit agent-written tests for time bombs. Absolute dates checked against rolling windows pass in review and fail on a schedule. Re-stamp fixture dates relative to now.
- Never let a worker close its own issue. Agents in our pipeline cannot mark their own work done; only the verifier can. That rule is why wave 10's problems were caught at all.
A night like this raises our confidence in the pipeline. It caught two real infrastructure problems overnight, attributed neither to the wrong party, and cost us one morning of cleanup. We have watched teams burn weeks on the same class of failure because their only judge was a human impression of "looks fine." If you want the fuller picture of how we check agent work, our report on AI agent evaluation covers the verification stack, and why AI agents fail covers the failure modes that are the model's fault.
