How do you let an agent experiment safely?
One worker agent edits a single Python file and states a hypothesis. A deterministic supervisor runs it in a sandbox against 100 test queries, measures accuracy and token cost, and keeps or discards the edit against thresholds calibrated before the night started.
Our knowledge base is a 14,000-chunk markdown vault searched by a hybrid retriever: vector similarity plus BM25 keyword scoring, fused by reciprocal rank. We froze a snapshot, wrote 130 test queries with hand-verified answer locations, and split them 100 for development and 30 hidden.
The overnight loop is two programs with a hard trust boundary between them. The worker, a Claude Sonnet agent, can only edit one file: the retrieval policy. The supervisor runs that policy in a sandbox with no network, checks whether the returned context still contains the verified answer for each query, and counts the tokens. The worker never writes any context text itself, so it cannot game the token count by paraphrasing.
The keep and discard thresholds were not vibes. Before the night, a calibration pass bootstrapped the query set and wrote the rules down: accuracy may not fall below 0.819, at most 3 previously-correct queries may flip to wrong, and an edit must cut at least 4.14% of tokens to replace the incumbent. The loop could not accept noise as improvement.
What did 60 trials find?
Five keeps, 56 discards. Four whole strategy families produced nothing. The one that paid: return only the sentences inside each retrieved chunk that share vocabulary with the query, and filter harder as chunks rank lower.
The worker organized its own search into five families: fusion changes, adaptive result counts, span selection, dedup, and query rewriting. Fusion went nowhere in eight tries, and dropping the vector half for pure keyword search cratered accuracy to 0.770. Global dedup kept deleting answer-bearing text. Twelve trials of query-side tweaks bought nothing on their own.
Span selection kept winning. Instead of returning whole retrieved chunks, return only the sentences that share vocabulary with the query. The refinement that mattered was rank-tiered strictness: the top two ranked chunks pass almost everything through, the third needs two distinct matching content words per sentence, and the fourth and fifth need three. Trust the head of the ranking, squeeze the tail.
The final policy: 199,992 tokens across the 100 dev queries, down from 258,863 at baseline, a 22.7% cut, with accuracy at 0.850 against the 0.880 baseline and inside every precommitted tolerance. The loop also found the cliff on its own. Two late trials pushed toward a 35% cut, accuracy fell through the floor, and the supervisor threw them out.
Did it survive queries it had never seen?
Yes, but a miscalibrated rollback rule fired first and called the night overfit. The paired comparison overturned it: identical hits to baseline on all 30 hidden queries at 18% fewer tokens, and better downstream answer quality.
Our predeclared morning rule checked the winner on the 30 hidden queries and fired a rollback: 0.733 accuracy, below the bar we had set from dev-set statistics. For an hour, the night was officially a failure.
Then we scored the unmodified baseline on the same 30 queries. Also 0.733. The hidden set was simply harder, and our rule had anchored its bar to dev-set difficulty instead of comparing the two policies on the same queries. Paired, the story flipped: the winner hit the exact same 22 of 30 queries as the baseline, lost zero, and spent 18% fewer tokens doing it.
We ran one more control. A fresh model answered the hidden questions twice, once from the winner's context and once from the baseline's, graded against hidden answer facts. The lean context scored 0.535 against 0.465, winning three questions outright and losing none. Cutting distractor text made answers better, not just cheaper.
The rollback rule was wrong and the result was right. Holdout comparisons must be paired: winner versus baseline on identical queries, never against a bar imported from a different query set.
What does this change in production?
The winning policy is 200 lines of dependency-free Python and now fronts every search our agents make against the knowledge base. Real queries return 5 to 10 KB of filtered context instead of whole files.
We ported the winner into a wrapper the same day. Our search engine still does the semantic ranking; the wrapper re-chunks the retrieved files, fuses in a local keyword ranking, keeps the top five chunks, and applies the rank-tiered sentence filter before anything reaches an agent's context window. Our agent instructions now point at the wrapper by default.
What transfers to your stack?
The recipe transfers. The exact percentage does not: 23% is a property of how much padding our chunks carry. Denser corpora will yield less.
- The waste in hybrid RAG usually lives inside retrieved chunks, not in the ranking. Tune span selection before you tune fusion.
- Trust decays with rank. Filter the tail of your retrieved set hard and leave the top results alone.
- Calibrate accept and reject thresholds from your own data before the loop runs, and write them down where the loop cannot edit them.
- Pair every holdout comparison. A bar derived from one query set will lie to you on another.
- A worker that edits one file plus a supervisor that measures two numbers is enough harness for 60 controlled experiments a night.
This is a single-corpus case study, run once, on one embedding model. We are not claiming a universal constant. We are claiming the harness is cheap, the failure modes are catchable, and the padding tax is probably sitting in your retriever too.
