Two AI Judges, One Document, Two Different Reviews

What Is LLM as a Judge?

LLM as a judge means using one language model to score another model's output against a rubric, in place of a human grader. It is fast, cheap and repeatable at volumes no person can read. It also carries biases a human grader does not, which is why the setup around it matters more than the model in it.

The pattern shows up in three jobs, and they are not equally safe: scoring model outputs in an evaluation suite, reviewing code or documents before a person reads them, and gating a pipeline that needs a pass or fail. We use it for all three. This report is where the line falls between them, measured on our own work rather than argued from first principles.

The published research already catalogs how automated judges go wrong. They favor whichever response is listed first, and swapping the order can flip the verdict (Wang et al., 2023). They favor longer answers, enough that a length correction became a standard fix (Dubois et al., 2024). They favor text that reads like their own output (Wataoka et al., 2024). The reference work on the method (Zheng et al., 2023) puts judge agreement with human preference at roughly the rate two humans agree, which is both the case for using judges and the reason to keep checking them.

Do Two AI Judges Review the Same Document the Same Way?

No. We handed two models an identical, self-contained brief on the same document, running in parallel, with neither able to see the other. One went after the evidence and re-ran our script. The other went after the shape of the argument and never opened the data. Same instructions, two different jobs.

Every measurement sat inline in the prompt, so neither judge read the repository and neither had memory of a previous round. Here is what came back, counted over the two committed reviews.

Average sentence length matches to a tenth of a word, so neither reviewer is padding. What differs is the target. Kimi carried two and a half times the numeric claims, Codex nearly three times the bullets. One audits the evidence, the other decomposes the structure.

The behavior matched the counts. Kimi re-ran our measurement script without being asked, compared its own output against the figures we had published, and found we had counted subagent text as human input. We published 14.4 percent. The correct figure was 10.6 percent. Codex never touched the data. It showed that our headline metric multiplied five rates measured over four different populations, so the arithmetic was right and the product meant nothing.

Then both of them opened with the same two objections. They ran in parallel and could not coordinate, so that agreement carried more weight than either review alone. It is what convinced us to rebuild the document instead of patching it.

Can Prompt Engineering Make an AI Judge Accurate?

Not on our data. We spent two days trying to turn one judge into a reliable pass or fail gate for specifications. It flagged 82 percent of specs when 35 percent were actually bad, and a constant approval that read nothing at all scored better than every prompt configuration we wrote.

The experiment was pre-registered, with a sealed holdout and an always-pass baseline. On synthetic extremes the judge was perfect. On real specifications it was a coin flip.

Then we pulled the lever everyone reaches for. We told it to be harsher. We told it to be gentler. We moved the stated base rate of failure and the stated cost of a miss. Every propensity lever moved the flag rate and not one improved the accuracy. Tilting it toward flagging produced nine false positives and zero true positives. Adding instructions made it monotonically worse, and one rewording collapsed it into approving everything.

The best configuration we found scored 44 out of 50 on the data we tuned it against, then caught zero of three real failures on the sealed holdout. It had memorized the phrasing of the visible examples rather than learning a mechanism.

We filed that as a failed experiment. It was the finding that makes the rest work. A reviewer you cannot trust with a verdict can still be excellent at producing findings. The model that lost to a rubber stamp on pass or fail is the same one that caught a real error in our published numbers. Two separate purchases, and we had been making them as one.

Why Does an AI Review Loop Never Finish?

Because the brief has no floor. Ask a capable model to find what is wrong with something and it will always return something, so the loop ends when the author runs out of patience. That makes patience the acceptance criterion. It is a missing rule rather than a difficult reviewer.

We read it as temperament for weeks. One judge stayed critical indefinitely, every round surfacing fresh and individually defensible objections, and no round was ever the last one. The diagnosis that fixed it is unglamorous: an open critique request has an unbounded output space, so the loop is unbounded by construction. Picking a gentler judge does not fix that, and overriding the judge on instinct quietly puts the decision back with the person who wrote the work.

What Makes an AI Review Loop Stop?

Four rules, decided before the first round. Findings must cite something. Findings are triaged by severity before any revision. Later rounds may only check whether earlier findings were resolved. Review stops on a round with nothing fatal in it, with a hard cap at three rounds.

• Citation gate. A finding that does not point at a line, a number, or a named inference is advisory by default. It is mechanically checkable, and it ends arguments about taste without having them.
• Severity triage, assigned before any revision. Fatal means a stated claim is false or a decision rests on an invalid inference, and it blocks acceptance. Material means the claim survives but its scope or confidence is wrong, so it gets qualified or recorded as a limitation. Advisory is logged as a ticket.
• No new territory after round one. Later rounds may only assess whether round one's findings were resolved. New ground becomes a ticket, not another revision cycle.
• Stopping conditions. A round with no new fatal findings is the normal exit. A hard cap of three rounds is the other: if fatal findings survive round three, revision is not fixing the problem, and the open items get escalated as they stand.

The third rule does the work. Without it, every round legitimately discovers something new, because any document is infinitely criticizable, and the sequence cannot converge. With it, the loop is bounded by round one's finding set, which is finite. Notice what the rules replace: the protocol never asks for a verdict, because the two days above told us the verdict is the part a judge is worst at.

Does It Mean Anything When Two AI Judges Agree?

Yes, when they genuinely could not see each other. Ours run in parallel and memoryless on the same self-contained prompt, so agreement is evidence rather than conformity. When both opened with the same two objections, that convergence justified a rebuild more than either critique would have alone.

Divergence carries information too, and it should not be averaged away. We record both readings and state which one we adopted, and when the disagreement would change a decision we decide it ourselves rather than letting the panel settle it by preference. In one recent round the judges split on the verdict line while agreeing on the substance: the same defect and the same recommended fix, filed at two severities. Two cautions from that round. Silence from a judge is not clearance, because a pass often means it examined a different question. And a judge's headline list can undercount its own findings, so tally from the body.

How Should You Set Up an LLM Judge Panel?

Draw the judges from different model families, ask for findings instead of verdicts, and write down what finishing looks like before the first round runs. A panel from one family reproduces that family's blind spots, and a panel with no stopping rule hands the acceptance decision to whoever tires first.

We do not anonymize the judges. The temptation is to strip identity so no reviewer gets weighted by reputation, but you do not anonymize a colleague, and the attribution is worth keeping because the two have different strengths. The citation gate already keeps their names out of the argument. Blinding would have been partial anyway: these models write distinguishably enough that we would infer the source from the prose, and closing that gap means having a model rewrite every finding into one schema, which is a place for criticism to get quietly softened.

The short version for a first panel. Use two model families, not two prompts. Require every finding to cite a line or a number. Decide in advance what makes a round the last one. Then let the rules make the acceptance call, rather than the reviewer's opinion or your own patience. It is the same discipline behind our agent evaluation work, and the reason our 154-trial model comparison was scored by two blind judges rather than one.