What does an open source eval harness actually give you?
It gives you the receipts instead of a leaderboard number. A task is a repo with a real bug, a verify script that fails before the fix and passes after it, and a prompt handed over verbatim. Every transcript, judgment, and token count lands in the repo next to the code.
Public benchmarks hand you a score and nothing to check it against. No transcript, no grading rubric, no way to tell whether a task was genuinely hard or just phrased to suit one vendor's house style. That gap is why we built model-eval and why it is now open. It runs a model headlessly against a small verifiable coding task, grades the diff with two independent judges, and writes the transcript, the judgment, and the raw token counts into the same repo the code lives in. No outcome depends on a number typed in by hand afterward.
- Every task proves its own difficulty. A self test shows the verify script failing on the unpatched repo, then passing once the reference fix is applied. A task that does not wire up cleanly fails CI rather than being skipped.
- Negative controls run the full prepare and grade path with no fix applied at all. Every one reports a failure, which is how we know the grader is not rubber stamping.
- Unattended runs sit inside an OS level sandbox profile. Credential reads are denied by default, writes stay in the run's own scratch tree, and API keys are popped from the child environment so every run is subscription authenticated.
- Two judges, one Claude and one Codex, score each diff on correctness, simplicity, idiomatic style, and spec adherence without seeing each other's verdict.
Two judges is a control, not decoration. Automated graders carry a documented self preference bias, so we checked the judges against each other before trusting either one. Across the 139 runs both judges scored, their per run averages land within a point of each other 95.7 percent of the time, and 518 of 556 individual dimension pairs agree within a point. Exact integer matches are rare, near 15 percent. Two independently prompted graders converging on the same rough tier but not the same integer is the expected shape, not a sign that one of them is noise.
Why does the same cost question give four different answers?
Because the word cheaper hides a choice about which configurations you compare. Pool every tier and the difference vanishes. Hold both models at medium and it stays gone. Hold them at high and the sign flips. Compare each model at its own cheapest passing tier and a real gap appears.
We compared Claude Fable 5 against GPT-5.6 Sol on the nine tasks both models solved, taking each model's median output tokens per solved run on each task. Output tokens only: all 64 of Fable's input token counts are quarantined as unreliable while all 112 of Sol's are measured, so a combined total would put an undercount against a true count and manufacture a gap out of a measurement artifact. Then we asked the same question four ways.
Read those four charts as one result. Pooled across tiers there is nothing. Matched at medium there is nothing. Matched at high the direction reverses. The only contrast that comes back significant compares Fable at medium against Sol at low, and those are different tiers, so some unknown part of that gap is the knob rather than the model. Fable's floor in this run matrix is medium, which means its cheapest passing configuration is an artifact of what we chose to run. Nine paired tasks is a coarse instrument regardless: the smallest two sided p this test can produce is 0.004, and that requires all nine tasks pointing the same way.
The narrow claim that survives is still useful. If you are picking a configuration to deploy on work in this difficulty band, Sol at low is the cheapest thing in this corpus that passes everything, at roughly a 1.6 times output token advantage over Fable at medium. Whether Fable at some lower tier would close that gap is a run we have not done.
Does turning the effort knob up buy anything?
On tasks in this difficulty band, no. Across 15 adjacent effort comparisons over 144 matched pairs, the number of runs that passed at one tier and failed at the next was zero. What the knob moved was spend. Sol at ultra costs about 4.4 times what Sol at low costs.
Every model here exposes an effort setting, and the pitch is that a higher setting buys better outcomes. The harness walks every adjacent pair of rungs per model, matching runs by task and repetition so task difficulty cancels out. Fifteen comparisons, 144 matched pairs, zero discordant results. Not rare, not statistically insignificant. Zero, for every model, turned in either direction.
| Model | low | medium | high | xhigh | max or ultra |
|---|---|---|---|---|---|
| GPT-5.6 Sol | 1,124 | 1,534 | 1,993 | 2,708 | 4,943 (ultra) |
| GPT-5.6 Luna | 1,551 | 2,874 | 5,249 (max) | ||
| GPT-5.3 Codex Spark | 2,308 | 3,530 | 4,250 | ||
| Kimi K3 | 1,385 | 1,631 | 1,972 (max) | ||
| Claude Fable 5 | 1,737 | 2,008 | |||
| Claude Haiku 4.5 | 2,861 | 2,675 | 2,745 (max) | ||
| Claude Haiku 4.5 (pinned) | 5,228 | 5,758 | 4,840 (max) |
What the knob moved was the bill. Sol at ultra costs about 4.4 times its low tier and Luna at max about 3.4 times its low, for no additional passing runs. Both Haiku rows step backwards, and neither has low as its cheapest tier, a reminder that the knob's name is a statement of intent and not a guarantee about behavior. That pattern matches published work on overthinking in reasoning models. The honest limit is the ceiling below: effort cannot buy a pass on a task that already passes at the floor.
Why did every model pass every task?
Because the tasks are too easy for current models, not because the models are equal. This is a ceiling effect. With 267 scorable runs spread over 25 cells, the Wilson lower bounds still run from 34 percent to 89 percent, so a perfect score proves considerably less than it looks.
Sol, Fable, Kimi K3, Luna, Codex Spark, and both Haiku snapshots cleared everything they attempted. One row of the 268 on disk is excluded as a CLI error, where the process died before the model got a fair attempt, and exclusions are printed with their counts at the top of the stats output rather than dropped in silence.
| Model | Cells | Runs | Passed | Best Wilson lower bound |
|---|---|---|---|---|
| GPT-5.6 Sol | 6 | 112 | 112 | 89% |
| Claude Fable 5 | 3 | 63 | 63 | 87% |
| GPT-5.6 Luna | 3 | 18 | 18 | 61% |
| GPT-5.3 Codex Spark | 3 | 18 | 18 | 61% |
| Kimi K3 | 3 | 18 | 18 | 61% |
| Claude Haiku 4.5 | 3 | 18 | 18 | 61% |
| Claude Haiku 4.5 (pinned) | 3 | 18 | 18 | 61% |
| Hybrid (harnessed) | 1 | 2 | 2 | 34% |
The design cost is the part worth stealing if you are building your own harness. At this many cells and this many runs per cell, the smallest true difference the design could detect is about 37 percentage points. A saturated benchmark stops discriminating long before its numbers start looking suspicious, and a perfect score sitting on a 34 percent lower bound is not evidence of much. The fix is harder tasks, not more runs, which is where the next round goes.
What happened when we got our own headline wrong?
The first draft led with a much better result. Same pass rate, roughly a fifth of the tokens, p equals 0.004. It was wrong. We had computed cost as input plus output tokens while every one of Fable's 64 input counts was already flagged unreliable in the data.
Input plus output is the obvious definition of cost and the wrong one here. We were reading a measurement gap as a performance gap. Worse, the repo's own stats script had been corrected for exactly this reason days earlier and states the output only rule in a comment. We did not run it. We recomputed by hand instead, and the hand rolled version quietly reintroduced the bug the script had been fixed to prevent.
Run the script and the flagship finding evaporates: p equals 0.68, direction mixed, nothing detectable. We left the correction in the record rather than restating the numbers quietly. A harness that asks you to trust its methodology should show you the time that methodology caught its author. The failure mode also generalizes past us. A result that is large, significant, and confirms what you already half believed is precisely the result nobody gives a second look. The wrong chart is still in the repo under a filename that says so, and it looks completely convincing.
How do you run this against your own tasks?
Clone the repo, run one task's self test offline, then run the harness in mock mode. Neither step spends a token or needs an API key. Point the runner at your own Claude Code or Codex CLI and it authenticates through the subscription you already pay for.
git clone https://github.com/drakegriffith/model-eval
cd model-eval
bash tasks/t1-py-a/selftest.sh # offline: verify.sh fails, then passes
python3 runner/run.py --mock --limit 1 # full pipeline, no tokens, no API keyAdding a task means writing a starting repo state with a real defect, a verify script, and a prompt. CONTRIBUTING.md covers that and the model registry. The harness is pointed at public models here because those are the ones people argue about, but the same structure is what we use on client systems, where the tasks are your workflows rather than someone's benchmark suite. That work lives in our agent development practice and our notes on evaluating agents.
