How do you let an agent experiment safely?
A worker agent may edit only one Python file, stating a hypothesis with each edit, and a deterministic supervisor judges every edit. It runs each one in a sandbox against 100 test queries, measures accuracy and token cost, and keeps or discards it against thresholds calibrated before the night.
Our knowledge base is a 14,000-chunk markdown vault searched by a hybrid retriever: vector similarity plus BM25 keyword scoring, fused by reciprocal rank. We froze a snapshot, wrote 130 test queries with hand-verified answer locations, and split them 100 for development and 30 hidden.
The two programs have a hard trust boundary between them. The worker, a Claude Sonnet agent, can edit only one file, the retrieval policy. The supervisor runs that policy in a sandbox with no network, checks whether the returned context still contains the verified answer for each query, and counts the tokens. The worker never writes any context text itself, so it cannot game the token count by paraphrasing.
The keep and discard thresholds came from our own data. Before the night, a calibration pass bootstrapped the query set and wrote the rules down: accuracy may not fall below 0.819, at most 3 previously-correct queries may flip to wrong, and an edit must cut at least 4.14% of tokens to replace the incumbent. With those margins, the loop could not accept noise as an improvement.
What did 60 trials find?
The loop kept four edits and discarded 56, and three entire strategy families produced nothing. The family that worked returns only the sentences inside each retrieved chunk that share vocabulary with the query, and it filters harder as chunks rank lower. Fusion changes went nowhere in eight tries.
The worker sorted its own search into five families: fusion changes, adaptive result counts, span selection, dedup, and query rewriting. Fusion went nowhere in eight tries, and switching to pure keyword search, without the vector half, cut accuracy to 0.770. Global dedup kept deleting answer-bearing text, and twelve trials of query-side tweaks produced no gain on their own.
Span selection kept winning: instead of returning whole retrieved chunks, it returns only the sentences that share vocabulary with the query. The refinement that mattered was rank-tiered strictness. The top two ranked chunks pass almost everything through, the third needs two distinct matching content words per sentence, and the fourth and fifth need three.
The final policy used 199,992 tokens across the 100 dev queries, down from 258,863 at baseline, a 22.7% cut, with accuracy at 0.850 against the 0.880 baseline and inside every precommitted tolerance. On its own, the loop also found the point where further cuts start losing answers. Two late trials pushed toward a 35% cut, accuracy fell below the floor, and the supervisor rejected them.
Did it survive queries it had never seen?
Yes, though a miscalibrated rollback rule fired first and called the night overfit. A paired comparison overturned it: the winner matched the baseline's hits on all 30 hidden queries at 18% fewer tokens, with better downstream answer quality. The hidden set was harder for both policies.
Our predeclared morning rule checked the winner on the 30 hidden queries and fired a rollback, because its 0.733 accuracy was below the bar we had set from dev-set statistics. For an hour we counted the run as a failure.
Then we scored the unmodified baseline on the same 30 queries, and it also got 0.733. The hidden set was harder, and our rule had anchored its bar to dev-set difficulty instead of comparing the two policies on the same queries. Compared query by query, the winner hit exactly the same 22 of 30 queries as the baseline, lost none, and used 18% fewer tokens.
We ran one more control. A fresh model answered the hidden questions twice, once from the winner's context and once from the baseline's, graded against hidden answer facts. The lean context scored 0.535 against 0.465, winning three questions outright and losing none, so removing distractor text made the answers better as well as cheaper.
The rollback rule rejected a result that was sound. A holdout comparison must pair the winner with the baseline on identical queries instead of measuring it against a bar imported from a different query set.
What does this change in production?
The winning policy is 200 lines of dependency-free Python and now fronts every search our agents make against the knowledge base. Real queries return 5 to 10 KB of filtered context instead of whole files. Our agent instructions point at the wrapper by default.
We ported the winner into a wrapper the same day. Our search engine still does the semantic ranking; the wrapper re-chunks the retrieved files, fuses in a local keyword ranking, keeps the top five chunks, and applies the rank-tiered sentence filter before anything reaches an agent's context window. Our agent instructions now point at the wrapper by default.
What transfers to your stack?
The method transfers, but the exact percentage will not, because the 23% reflects how much padding our chunks carry and denser corpora will yield less. This was one corpus and one embedding model, run once. If you try it, tune span selection before fusion, and pair every holdout comparison.
- The waste in hybrid RAG usually sits inside the retrieved chunks rather than in the ranking, so tune span selection before you tune fusion.
- Lower-ranked chunks deserve less trust, so filter the tail of your retrieved set hard and leave the top results alone.
- Calibrate accept and reject thresholds from your own data before the loop runs, and write them down where the loop cannot edit them.
- Pair every holdout comparison, because a bar derived from one query set will mislead you on another.
- A worker that edits one file and a supervisor that measures two numbers are enough to run 60 controlled experiments in a night.
This is a single-corpus case study, run once, on one embedding model, so we are not offering 23% as a universal figure. We do claim that the harness is cheap, its failure modes can be caught, and your retriever is probably carrying the same padding.
