What Are AI Benchmarks?
An AI benchmark is a fixed set of tasks paired with an answer key, used to score and rank models. The score depends on three things people rarely separate: the difficulty of the tasks, the correctness of the key, and the grader that decides whether a given answer counts as a match.
Those three parts fail independently. A benchmark can hold genuinely hard tasks and still report noise, because its answer key contains errors or because its grader is lenient. When a leaderboard shows two models three points apart, that gap can come from either. Nothing on the page tells you which.
The distinction matters most at purchase time. A vendor quoting a benchmark score is quoting a measurement made under conditions you cannot see. Below is what happened when two teams went and looked.
How Often Is the Benchmark Answer Key Itself Wrong?
Often enough to change the ranking. Penfield Labs audited LoCoMo, a widely cited memory benchmark, and found 99 of its 1,540 answers were wrong, or 6.4 percent. Because the key is broken, a perfect system scores about 93.6 percent, and small reported gaps between systems stop meaning anything.
The errors were not typos. The audit found answers referencing facts the conversation never contained, dates resolved to the wrong day of the week, and 24 questions attributing a statement to the wrong speaker. Every memory system ranked on LoCoMo was graded against those answers, and the flaw sat in public while vendors published scores against it.
A benchmark's key is data like any other data, and it goes uninspected for the same reason production data does. Nobody is assigned to check it.
Can an AI Model Be Trusted to Grade Another Model?
Not without validation you can inspect. LoCoMo does not match answers to its key as text. It asks gpt-4o-mini whether an answer is close enough. Auditors fed that grader deliberately wrong answers that stayed on topic, and it passed 62.81 percent of them.
That result means a plausible wrong answer scored about as well as a correct one, so the leaderboard was partly measuring how much vagueness one small model would tolerate. We have measured the same instability in our own work: two AI judges given a byte-identical brief on one document differed by 2.5 times on numeric claims and 2.8 times on bullet points. The full write-up is in our report on LLM as a judge.
Model graders are not disqualified by this. They are cheap and they scale. They need a disclosed validation step, a second independent grader, and a published agreement rate, and most benchmarks ship none of the three.
Why Did OpenAI Retract a Benchmark It Recommended?
Because its own audit found the tasks broken. OpenAI had pointed the community at SWE-Bench Pro after retiring SWE-bench Verified for contamination. Its analysis pipeline then flagged 200 of the 731 public tasks, 27.4 percent, and five engineers reviewing by hand flagged 249, or 34.1 percent.
The four failure categories OpenAI published are worth memorizing, because they recur in private evaluations too: tests so strict they reject functionally correct work, prompts that omit requirements the hidden tests enforce, tests too thin to catch an incomplete fix, and prompts that point at the wrong behavior outright.
On that same split, frontier models climbed from a 23.3 percent pass rate to 80.3 percent in eight months. How much of that 57 point rise is coding ability and how much is broken tasks is not something the number can tell you.
| Benchmark | Audited by | Share found broken | Effect on the score |
|---|---|---|---|
| LoCoMo, 1,540 answers | Penfield Labs | 99 answers, 6.4% | A perfect system caps near 93.6% |
| LoCoMo grader | Penfield Labs | 62.81% of wrong answers passed | Plausible wrong answers score like correct ones |
| SWE-Bench Pro, 731 tasks | OpenAI analysis pipeline | 200 tasks, 27.4% | Recommendation retracted |
| SWE-Bench Pro, 731 tasks | Five engineers, by hand | 249 tasks, 34.1% | Human review found more than the pipeline |
What Happens When Every Model Passes?
The benchmark stops discriminating and you need a different axis. Across 267 scorable runs in our own coding harness, every model passed every task it attempted. Pass rate had saturated, so the signal moved to cost, consistency, and judged quality instead.
Saturation is the quiet failure mode. A benchmark that everything passes looks healthy and reports nothing. When we ran 154 controlled trials pitting Claude against GPT, both models passed everything solvable and there was no detectable difference to report, which is the honest finding rather than a disappointing one. Details are in our open source eval harness.
Consistency is the axis most demos hide. On tau-bench, gpt-4o succeeded on fewer than half the tasks, and its pass rate over eight repeated attempts of the same task dropped below 25 percent in retail. Same model, same tasks. A demo shows you the first number and production charges you for the second.
How Do You Check Whether a Benchmark Is Reproducible?
Try to run it. Open the reproduction section and confirm the harness, the datasets, and the grader prompt are actually published. Many benchmark documents list commands pointing at directories that never shipped, which makes every number in the file unverifiable by anyone outside the team.
This check takes about five minutes and it is the highest yield thing you can do with a benchmark claim. Three passes, in order:
- Clone the repository named in the write-up and look for the harness directory that the reproduction commands actually reference.
- Confirm the answer key and the grader prompt ship in the tree, rather than being described in prose.
- Check whether a judge agreement rate is published, and whether a second independent grader produced it.
If any of the three comes back empty, the score is a claim rather than a measurement. That verdict is not an accusation of bad faith. Publishing a harness is a week of unglamorous work that wins no attention, so skipping it is the default, and the default is invisible.
Self-graded and auditable is a different thing from self-graded and unauditable. Nearly every number you will read was produced by a team with a stake in the outcome, so independence is the wrong test. Reproducibility is the test that survives, and the good projects pass it: they ship the fixture generator, the corpus seed, the held-out set, and the agreement rate between graders.
Which AI Benchmarks Should You Use for Your Own Decision?
Use public benchmarks to rule models out, never to pick one. They measure a general model on generic tasks, not your system on your data. For a buying decision, build a small evaluation from your own real tasks with a command-checkable definition of done.
We run public benchmarks as a coarse filter and then stop. The decision gets made on a private set of tasks drawn from the work a client actually needs done, scored by a check that either exits zero or does not. Twenty real tasks graded by a script beat a public leaderboard for the same reason a road test beats a brochure.
The approach generalizes past model selection. Any AI system you deploy needs a verifier separate from the thing being verified, and that gate has to read its rule from somewhere the worker cannot edit. That principle is the backbone of how we build, and it is why we can tell you what our own systems fail at.
