How Does the Overnight Pipeline Work?
GitHub issues run in overnight waves through a sandboxed loop, where an LLM agent works each issue serially on a wave branch inside Docker and git worktree isolation. A bash script judges each result with ruff lint against a baseline, ruff format on changed files, the full pytest suite, and a green GitHub Actions run on the exact branch tip.
The design rests on one rule, which is that the checker is never the worker. The worker is an LLM, so we made the judge a bash script on purpose. It runs ruff lint against a recorded baseline, ruff format on the files the agent changed and the full pytest suite, and then requires a green GitHub Actions run on the exact commit at the branch tip. The issue stays open unless local checks and CI are both green on the same sha.
A script this dumb cannot be argued with, because it does not reason. An LLM judge can be talked into a pass by an LLM worker, or can talk itself into one, while an exit code cannot be talked into anything. When the judge says red, something is red, and the work is finding out what.
What Happened When Wave 10 Came Back Red?
All three issues in wave 10, two test-isolation fixes and one dashboard feature, came back red. The obvious reading was that the agents had written bad code overnight, but the logs showed the failures sitting in files that no agent in the wave had touched.
Wave 10 was small, with two test-isolation fixes and one dashboard feature. When a wave that ordinary goes fully red, usually either the agents had a bad night or something under them moved. Our first instinct was to blame the agents, and we expect most people's would be the same.
The verifier log pointed elsewhere: the format gate had failed on 18 files, and none of the three issues in the wave had touched any of them.
Why Did 18 Files the Agents Never Edited Fail Formatting?
The 18 files failed because the format gate had just started working after weeks of passing without checking anything, broken by a bash 3.2 portability bug on macOS. The previous wave fixed that bug, so the gate's first working run caught all the formatting debt that had built up on trunk over those weeks.
The format gate in our verifier had been broken for weeks. The script used bash syntax that macOS's old bash 3.2 does not support, so on our hardware the gate exited green without checking a single file. We missed it for that long because a gate that always passes looks the same from the outside as a codebase that is always clean.
The previous wave had fixed that portability bug as part of unrelated hardening, which made wave 10 the first wave judged by a format gate that worked. Its first working run found all 18 files that had drifted while the gate was broken, and the agents in wave 10 had touched none of them.
Weeks of format drift surfaced on the gate's first working run.
We paid that debt down and ran the verifier again, and the wave was still red.
Why Were Two Tests Out of 3,354 Still Failing?
Two tests out of 3,354 were still failing because their fixture used absolute dates, July 11 through 13, which the assertion checked against a rolling 100-hour freshness window. The tests passed one day and failed the next because the calendar moved. A helper that re-stamps fixture dates relative to now fixed them for good.
The failing tests checked a freshness rule that counts data as recent if it falls inside a rolling 100-hour window. The fixture feeding them used absolute dates, July 11 through July 13, written in as literals. On the day the tests were written those dates sat well inside the window, but the window kept moving and the dates did not.
The tests passed on the day they were merged, passed the next day, and failed the day after that with no code change anywhere. An agent had written that fixture weeks earlier, and its dates happened to age out of the window during wave 10. The fix was a small helper that re-stamps fixture dates relative to the current time, so the data is always the same age whenever the suite runs.
Agent-written tests tend to use concrete example data, and concrete dates go stale. If your agents write tests, grep the fixtures for date literals now instead of waiting for one of them to expire.
So Were the Agents Ever Guilty?
No. The agent-written changes in wave 10 were fine as submitted. One red came from a gate that had started working and the other from a stale fixture already on trunk. Both looked like agent failures at first glance, which is why we let a script decide pass or fail instead of a person's first impression.
Once the debt was paid and the fixture re-stamped, wave 10's work passed every gate without changes. Every red that night came either from the harness getting stricter or from old debt on trunk that happened to surface at the same time.
Where wave 10's red actually came from
Source: Wave 10 verifier logs, 2026-07-15| Red | Looked like | Turned out to be | The fix |
|---|---|---|---|
| First verifier run | Agents shipped unformatted code | 18 files of trunk debt, exposed by the format gate's first working run | Format the backlog, keep the gate |
| Second verifier run | Agents broke two tests | Absolute fixture dates aged past a rolling 100-hour freshness window | Re-stamp fixture dates relative to now |
Twice in one night we read "red wave" as "bad agents," and both times the evidence pointed at our own infrastructure. We think the bias is worth naming because it decides where the debugging hours go. Our report on harness engineering makes the general case that when an agent fails, you should audit the harness before you blame the model, and wave 10 was a direct example of it.
What Should You Steal From This?
We came away with six rules: fix the harness before you blame the model, keep the judge too dumb to persuade, require green locally and in CI on the same commit, expect new gates to surface old debt, audit agent-written tests for hard-coded dates, and never let a worker close its own issue.
- Fix the harness before the model. Both failures were in our infrastructure, and neither would have been fixed by a better prompt or a bigger model.
- Keep the judge dumb. A bash script cannot be persuaded, and because our verifier could not be talked out of a red, we trusted it enough to keep digging instead of overriding it.
- Require two green signals, local checks and a CI run on the exact branch tip. The bash 3.2 bug shows why, since our local gate reported green for weeks without checking anything.
- Expect a new gate to surface old debt on day one. The first working run of any checker reports everything that drifted while it was broken, so budget time for that and do not read it as a regression.
- Audit agent-written tests for date time bombs. Absolute dates checked against a rolling window pass review and fail once enough time passes, so re-stamp fixture dates relative to now.
- Never let a worker close its own issue. Agents in our pipeline cannot mark their own work done, and only the verifier can, which is why wave 10's problems were caught at all.
A night like this raises our confidence in the pipeline, because it caught two real infrastructure problems overnight, attributed neither to the wrong party, and cost us one morning of cleanup. We have seen teams spend weeks on the same class of failure because their only judge was a person deciding it "looks fine." Our report on AI agent evaluation covers the verification stack we use to check agent work, and why AI agents fail covers the failure modes that are the model's fault.
