What does your own eval harness actually give you?
It gives you the full record behind each result instead of a single leaderboard number. A task is a repo with a real bug, a verify script that fails before the fix and passes after it, and a prompt handed over verbatim. Every transcript, judgment, and token count lands in the repo next to the code.
Public benchmarks give you a score and nothing to check it against: no transcript, no grading rubric, and no way to tell whether a task was actually hard or just phrased to suit one vendor's house style. That gap is why we built model-eval for our own testing. It runs a model headlessly against a small verifiable coding task, grades the diff with two independent judges, and writes the transcript, the judgment and the raw token counts into the same repo the code lives in, so no outcome depends on a number typed in by hand afterward.
- Every task proves its own difficulty. A self test shows the verify script failing on the unpatched repo, then passing once the reference fix is applied. A task that does not wire up cleanly fails CI rather than being skipped.
- Negative controls run the full prepare and grade path with no fix applied at all. Every one reports a failure, which is how we know the grader is not rubber stamping.
- Unattended runs sit inside an OS level sandbox profile. Credential reads are denied by default, writes stay in the run's own scratch tree, and API keys are popped from the child environment so every run is subscription authenticated.
- Two judges, one Claude and one Codex, score each diff on correctness, simplicity, idiomatic style, and spec adherence without seeing each other's verdict.
The second judge is there as a control. Automated graders carry a documented self preference bias, so we checked the judges against each other before trusting either one. Across the 139 runs both judges scored, their per run averages land within a point of each other 95.7 percent of the time, and 518 of 556 individual dimension pairs agree within a point. Exact integer matches are rare, near 15 percent. That is what we would expect from two independently prompted graders that agree on the rough tier without landing on the same integer, and it does not mean one of them is noise.
Why does the same cost question give four different answers?
Because the word cheaper hides a choice about which configurations you compare. Pooled across every tier, the two models show no detectable difference, and holding both at medium effort shows none either. Held at high, the sign flips. Only when each model runs at its own cheapest passing tier does a real gap appear.
We compared Claude Fable 5 against GPT-5.6 Sol on the nine tasks both models solved, taking each model's median output tokens per solved run on each task. We counted output tokens only, because all 64 of Fable's input token counts are quarantined as unreliable while all 112 of Sol's are measured, and a combined total would set an undercount against a true count and create a gap out of a measurement artifact. Then we asked the same question four ways.
The four charts should be read together. Pooled across tiers and matched at medium there is no detectable difference, and matched at high the direction reverses. The only contrast that comes back significant compares Fable at medium against Sol at low, and those are different tiers, so some unknown part of that gap is the knob rather than the model. Fable's floor in this run matrix is medium, which means its cheapest passing configuration is an artifact of what we chose to run. Nine paired tasks is a coarse instrument in any case: the smallest two sided p this test can produce is 0.004, and only if all nine tasks point the same way.
A narrower claim does survive, and it is still useful. If you are picking a configuration to deploy on work in this difficulty band, Sol at low is the cheapest thing in this corpus that passes everything, with roughly a 1.6 times output token advantage over Fable at medium. We have not run Fable at a lower tier, so we do not know whether that would close the gap.
Does turning the effort knob up buy anything?
On tasks in this difficulty band, no. Across 15 adjacent effort comparisons over 144 matched pairs, no run passed at one tier and failed at the next. What the knob did change was spend, with Sol at ultra costing about 4.4 times what Sol at low costs.
Every model here exposes an effort setting, and the pitch is that a higher setting buys better outcomes. The harness walks every adjacent pair of rungs per model, matching runs by task and repetition so task difficulty cancels out. Across 15 comparisons and 144 matched pairs there were zero discordant results, in either direction, for every model.
| Model | low | medium | high | xhigh | max or ultra |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 1,124 | 1,534 | 1,993 | 2,708 | 4,943 (ultra) |
| GPT-5.6 Luna | 1,551 | 2,874 | 5,249 (max) | ||
| GPT-5.3 Codex Spark | 2,308 | 3,530 | 4,250 | ||
| Kimi K3 | 1,385 | 1,631 | 1,972 (max) | ||
| Claude Fable 5 | 1,737 | 2,008 | |||
| Claude Haiku 4.5 | 2,861 | 2,675 | 2,745 (max) | ||
| Claude Haiku 4.5 (pinned) | 5,228 | 5,758 | 4,840 (max) |
The knob did move the bill. Sol at ultra costs about 4.4 times its low tier and Luna at max about 3.4 times its low, with no additional passing runs. Both Haiku rows step backwards, and neither has low as its cheapest tier, so a tier's name states an intent and does not guarantee how the model behaves. That pattern matches published work on overthinking in reasoning models. All of this is limited by the ceiling described below, since effort cannot buy a pass on a task that already passes at the floor.
Why did every model pass every task?
Because the tasks are too easy for current models, which is a ceiling effect and says nothing about whether the models are equal. With 267 scorable runs spread over 25 cells, the Wilson lower bounds still run from 34 percent to 89 percent, so a perfect score proves considerably less than it looks.
Sol, Fable, Kimi K3, Luna, Codex Spark, and both Haiku snapshots cleared everything they attempted. One row of the 268 on disk is excluded as a CLI error, where the process died before the model got a fair attempt, and exclusions are printed with their counts at the top of the stats output rather than dropped in silence.
| Model | Cells | Runs | Passed | Best Wilson lower bound |
|---|---|---|---|---|
| GPT-5.6 Sol | 6 | 112 | 112 | 89% |
| Claude Fable 5 | 3 | 63 | 63 | 87% |
| GPT-5.6 Luna | 3 | 18 | 18 | 61% |
| GPT-5.3 Codex Spark | 3 | 18 | 18 | 61% |
| Kimi K3 | 3 | 18 | 18 | 61% |
| Claude Haiku 4.5 | 3 | 18 | 18 | 61% |
| Claude Haiku 4.5 (pinned) | 3 | 18 | 18 | 61% |
| Hybrid (harnessed) | 1 | 2 | 2 | 34% |
If you are building your own harness, the part worth copying is the check on what the design can detect. At this many cells and this many runs per cell, the smallest true difference the design could detect is about 37 percentage points. A saturated benchmark stops discriminating long before its numbers start looking suspicious, and a perfect score sitting on a 34 percent lower bound is weak evidence. The fix is harder tasks rather than more runs, and that is where the next round goes.
What happened when we got our own headline wrong?
We retracted it and kept the correction on the record. The first draft led with a much better result, the same pass rate at roughly a fifth of the tokens (p equals 0.004), but we had computed cost as input plus output tokens while every one of Fable's 64 input counts was already flagged unreliable.
Input plus output is the obvious definition of cost, but it was the wrong one here, and we were reading a measurement gap as a performance gap. The repo's own stats script had been corrected for exactly this reason days earlier and states the output only rule in a comment. We recomputed by hand instead of running it, and the hand rolled version reintroduced the bug the script had been fixed to prevent.
Running the script makes the flagship finding disappear: p equals 0.68, the direction is mixed, and nothing is detectable. We left the correction in the record instead of silently restating the numbers, because a harness that asks you to trust its methodology should show you the time that methodology caught its author. The same mistake is easy for anyone to make, since a result that is large, significant and confirms what you already half believed is exactly the result nobody gives a second look. The wrong chart is still in the repo under a filename that says so, and it looks completely convincing.
How would you build a harness like this for your own tasks?
Start with one task: a starting repo state with a real defect, a verify script that fails before the fix and passes after it, and a prompt handed over verbatim. Prove that wiring with an offline self test, then run the whole pipeline in a mock mode that spends no tokens before pointing it at a real model.
In our harness, adding a task means writing those three pieces and adding the model to a registry. The harness itself is not public, so treat this section as a description of the design rather than a download. It is pointed at public models here because those are the ones people argue about, but the same structure is what we use on client systems, where the tasks are your workflows rather than someone's benchmark suite. That work lives in our agent development practice and our notes on evaluating agents.
