What Are AI Benchmarks?
An AI benchmark is a fixed set of tasks paired with an answer key, used to score and rank models. The score depends on three things people rarely separate: the difficulty of the tasks, the correctness of the key, and the grader that decides whether a given answer counts as a match.
Those three parts fail independently. A benchmark can hold hard tasks and still report noise because its answer key contains errors or its grader is lenient. When a leaderboard shows two models three points apart, the gap can come from either, and nothing on the page tells you which.
This matters most when you are buying, because a vendor quoting a benchmark score is quoting a measurement made under conditions you cannot see. The sections below cover what two teams found when they went and looked.
How Often Is the Benchmark Answer Key Itself Wrong?
Often enough to change the ranking. Penfield Labs audited LoCoMo, a widely cited memory benchmark, and found 99 of its 1,540 answers were wrong, or 6.4 percent. Because the key is broken, a perfect system scores about 93.6 percent, so small reported gaps between systems stop meaning anything.
The errors were not typos. The audit found answers referencing facts the conversation never contained, dates resolved to the wrong day of the week, and 24 questions attributing a statement to the wrong speaker. Every memory system ranked on LoCoMo was graded against those answers, and the flaws sat in public while vendors published scores against them.
A benchmark's key is data like any other, and it goes uninspected for the same reason production data does: nobody is assigned to check it.
Can an AI Model Be Trusted to Grade Another Model?
Not without validation you can inspect. Instead of matching answers to its key as text, LoCoMo asks gpt-4o-mini whether an answer is close enough, and when auditors fed that grader deliberately wrong answers that stayed on topic, it passed 62.81 percent of them.
That result means a plausible wrong answer scored about as well as a correct one, so the leaderboard was partly measuring how much vagueness one small model would tolerate. We have measured the same instability in our own work: two AI judges given a byte-identical brief on one document differed by 2.5 times on numeric claims and 2.8 times on bullet points. The full write-up is in our report on LLM as a judge.
Model graders are still useful because they are cheap and they scale, but they need a disclosed validation step, a second independent grader and a published agreement rate, and most benchmarks ship none of the three.
Why Did OpenAI Retract a Benchmark It Recommended?
Because its own audit found the tasks broken. OpenAI had pointed the community at SWE-Bench Pro after retiring SWE-bench Verified for contamination. Its analysis pipeline then flagged 200 of the 731 public tasks, 27.4 percent, and five engineers reviewing by hand flagged 249, or 34.1 percent.
The four failure categories OpenAI published are worth memorizing, because they recur in private evaluations too: tests so strict they reject functionally correct work, prompts that omit requirements the hidden tests enforce, tests too thin to catch an incomplete fix, and prompts that point at the wrong behavior outright.
On that same split, frontier models climbed from a 23.3 percent pass rate to 80.3 percent in eight months, and the number alone cannot tell you how much of that 57 point rise is coding ability and how much is broken tasks.
| Benchmark | Audited by | Share found broken | Effect on the score |
|---|---|---|---|
| LoCoMo, 1,540 answers | Penfield Labs | 99 answers, 6.4% | A perfect system caps near 93.6% |
| LoCoMo grader | Penfield Labs | 62.81% of wrong answers passed | Plausible wrong answers score like correct ones |
| SWE-Bench Pro, 731 tasks | OpenAI analysis pipeline | 200 tasks, 27.4% | Recommendation retracted |
| SWE-Bench Pro, 731 tasks | Five engineers, by hand | 249 tasks, 34.1% | Human review found more than the pipeline |
What Happens When Every Model Passes?
The benchmark stops discriminating and you need a different axis. Across 267 scorable runs in our own coding harness, every model passed every task it attempted. Pass rate had saturated, so the signal moved to cost, consistency, and judged quality instead.
Saturation is easy to miss, because a benchmark that every model passes looks healthy while reporting nothing. When we ran 154 controlled trials pitting Claude against GPT, both models passed everything solvable and there was no detectable difference to report, which is a finding in its own right rather than a disappointing one. Details are in our eval harness write-up.
Most demos hide consistency. On tau-bench, gpt-4o succeeded on fewer than half the tasks, and in retail its pass rate over eight repeated attempts of the same task dropped below 25 percent. A demo shows you the first number, and in production you live with the second.
How Do You Check Whether a Benchmark Is Reproducible?
Try to run it. Open the reproduction section and confirm the harness, the datasets, and the grader prompt are actually published. Many benchmark documents list commands pointing at directories that never shipped, which makes every number in the file unverifiable by anyone outside the team.
This check takes about five minutes and it is the most useful thing you can do with a benchmark claim. Run these three steps in order:
- Clone the repository named in the write-up and look for the harness directory that the reproduction commands actually reference.
- Confirm the answer key and the grader prompt ship in the tree, rather than being described in prose.
- Check whether a judge agreement rate is published, and whether a second independent grader produced it.
If any of the three comes back empty, the score is a claim rather than a measurement. That does not imply bad faith: publishing a harness is a week of unglamorous work that wins no attention, so skipping it is the norm and nobody notices.
A self-graded benchmark you can audit and one you cannot are different things. Nearly every number you will read was produced by a team with a stake in the outcome, so independence is the wrong test, and reproducibility is the one that still works. The good projects pass it by shipping the fixture generator, the corpus seed, the held-out set and the agreement rate between graders.
Which AI Benchmarks Should You Use for Your Own Decision?
Use public benchmarks to rule models out, never to pick one. They measure a general model on generic tasks, while your result depends on your own system and your own data. For a buying decision, build a small evaluation from your own real tasks with a command-checkable definition of done.
We run public benchmarks as a coarse filter and then stop. The decision gets made on a private set of tasks drawn from the work a client actually needs done, scored by a check that either exits zero or does not. Twenty real tasks graded by a script are a better guide than a public leaderboard, in the way a road test is a better guide than a brochure.
The same approach applies beyond model selection. Any AI system you deploy needs a verifier separate from the thing being verified, and that gate has to read its rule from somewhere the worker cannot edit. We build around that principle, and it is why we can tell you what our own systems fail at.
