We Ran 154 Controlled Trials of Claude vs GPT. Both Scored Perfect.

Why Do Most Claude Versus GPT Comparisons Tell You Nothing?

Single-run comparisons cannot survive non-determinism. Ask the same model the same question twice and you can get two different answers, so one run tells you what happened once, not what happens on average. A fair comparison needs repetition, identical conditions, and honest statistics.

The problem is non-determinism. Ask the same model the same coding question twice and you can get two different answers, two different token counts, two different outcomes. A single run tells you what happened once. It tells you nothing about what happens on average, which is the only thing you actually care about when you are deciding which tool your team ships with. Run the comparison again tomorrow and the winner can flip.

There is a second, quieter problem. The public benchmarks that do use repeated, structured evaluation are running out of headroom. Frontier models now cluster near the top of the standard suites, which makes those suites worse at telling models apart. Contamination compounds it: when benchmark problems, or text closely derived from them, land in a model's training data, the model can recall an answer instead of reasoning to it. The team behind LiveCodeBench (Jain et al., 2024) documented exactly this for coding, showing that older suites are no longer sufficient on their own and building a time-segmented benchmark of fresh contest problems specifically to dodge the leakage. We said the same thing in our own terms in our report on AI agent evaluation: public benchmarks measure the model, not your system.

How Do You Design a Fair Test of Two AI Coding Models?

We ran a full-factorial designed experiment blocked by task: model by reasoning effort by harness, every configuration facing every one of nine tasks, each task proven solvable before any model touched it. Runs were seed-interleaved and every outcome scored by a deterministic pass or fail gate.

The two contestants were Claude Fable 5, driven through Anthropic's claude CLI, and GPT-5.6, driven through OpenAI's codex CLI. The design is a full-factorial designed experiment, blocked by task. The factors are model (Claude, GPT), reasoning effort (two to three levels), and harness (a bare CLI versus a shared agent-instruction harness). Blocking means every configuration sees every task, so task difficulty cancels out of any paired comparison. Replication is n=3 repetitions per cell on the smaller tasks and n=2 on the greenfield build, because a single run is an anecdote and repeated runs let us treat each outcome as a Bernoulli trial and estimate a real pass rate. Run order is seed-interleaved (seed 1337) so that time-of-day and API-load drift cannot systematically favor one model.

The tasks matter as much as the design. There are nine, and every one was proven solvable before any model touched it. Four are seeded-bug tasks with deliberately subtle defects: an off-by-one in pagination, a naive-versus-timezone-aware datetime bug, a cache key that silently drops a dimension, and an async dedup race. Four are feature tickets written in a strict six-section format. One is a greenfield CLI build, a roughly 200-line cost-splitting tool with a reference implementation. Each task ships a self-test that fails on the broken starting code and passes after the reference patch. All nine were green before the experiment began, so a failure would mean the model failed, not that the task was impossible.

Conditions were held byte-identical: the same harness instruction files, the same prompts, and the same deterministic verify gate deciding pass or fail. Every transcript and diff was then stripped of any identifying markers and scored blind by two separate judges, a Claude judge and a GPT judge, on a four-dimension rubric. We ran two judges precisely because a single-model judge is a known failure mode, which we come back to below. Statistics are exact, not large-sample approximations, because the per-cell sample sizes are small and pretending otherwise would be dishonest: Wilson score intervals for every pass rate, McNemar's exact test for paired head-to-heads, an exact sign-flip permutation test on tokens, and an explicit power statement for what the sample can detect.

What Did 154 Trials Actually Show?

Both models passed everything solvable. Sweep 1 was 120 of 120; every solo configuration across both sweeps was 152 of 152. The paired McNemar test found zero discordant pairs, so there was no detectable difference between the models on this suite.

Sweep 1 was 120 passes out of 120 attempts, across both models and every effort level. Fold in the solo runs from the second sweep, the harnessed tasks and the bare-plus-harnessed greenfield build, and the total is 152 of 152 clean passes on every configuration where a model worked alone. The honest way to report a perfect score is with its confidence interval, not as a bare 100 percent. Twenty-four passes out of twenty-four gives a 95 percent Wilson interval of 86.2 percent to 100 percent. That is the correct reading: we did not prove the model never fails on tasks like these. We proved that if it has a failure rate, the data are consistent with it being anywhere from zero up to about 14 percent. The chart at the top of this report draws every perfect bar with exactly that uncertainty. A perfect sample is evidence, not a guarantee, and the interval is where the honesty lives.

For the head-to-head we used McNemar's exact test, which only extracts information from discordant pairs, the cases where one model passed and the other failed on the same task and repetition. There were zero discordant pairs. Every matched pair was pass and pass. With nothing discordant, there is nothing to test, and the correct conclusion is no detectable difference between the models on this suite. Not that they are equal. No detectable difference. Those are different claims, and we are careful to only make the one the data supports.

One configuration did wobble, and it deserves the careful version rather than a headline. We also tested a hybrid: Claude orchestrating GPT as the implementer. Across four attempts, two produced planning notes and zero code, and two shipped working solutions of roughly 160 to 176 lines. So, 2 of 4. The rig was identical across all four, which means the most defensible reading is genuine run-to-run variance in a multi-agent handoff, not a fixed defect. With n=4, that is where we stop. It is a flag for the next experiment, not a conclusion.

Does Higher Reasoning Effort Improve Code Quality?

No, not on this suite. Pass rate stayed at 100 percent at every effort level for both models. The only thing that moved was cost: GPT-5.6 at high effort burned about 80 percent more output tokens than at low, for the identical result.

Every model here exposes a reasoning-effort setting, and the pitch is that higher effort makes the model think harder and do better. On this suite, it did not. Pass rate was 100 percent at every effort level for both models. What moved was cost. Claude at high effort spent about 16 percent more output tokens than at medium (2,008 versus 1,736 median) for the identical result. GPT-5.6 at high effort spent about 80 percent more output tokens than at low (1,966 versus 1,095) and pulled roughly 1.8 times the input tokens (174.7k versus 95.2k). The pairwise McNemar tests between adjacent effort rungs again found zero discordant pairs. More effort, same outcome, higher bill. That is the same routing discipline we cover in our report on token efficiency: spend the expensive setting only where it changes an outcome.

This is not a quirk of our nine tasks. It matches a growing body of work on what researchers call the overthinking phenomenon. The survey Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (Sui et al., 2025) catalogs how reasoning models generate verbose, redundant chains that add computational cost without adding correctness. The empirical study Between Underthinking and Overthinking (Su et al., 2025) is sharper still: models overthink easy problems, producing unnecessarily long outputs, precisely because they are poor at gauging problem difficulty. Our tasks were, in hindsight, easy for these models. Cranking the effort dial on an easy task is spending money to watch a model deliberate over a conclusion it already had.

Do Claude and GPT Solve Problems the Same Way?

No. Claude iterates over six to seven turns, reading, editing, and checking until it converges. GPT one-shots in a single turn. Same destination, different route, and neither style won on outcomes because both cleared this tier.

The two models do not solve problems the same way, and the trace makes it obvious. Claude iterates. Median 6 to 7 turns per task, roughly 59 to 68 seconds of wall-clock time. It reads, edits, checks, and edits again, converging on a fix. GPT one-shots. Median of a single turn, roughly 26 to 64 seconds. It reasons internally and emits a solution in one pass. Neither style won on outcomes at this tier, because both styles cleared the tier. If you have watched one of these tools and formed an opinion about which works harder, this is a useful reminder that turn count is a workflow signature, not a quality signal.

One caveat you cannot skip: input tokens do not compare across vendors. Look back at the config table and you will see Claude reporting around 19.4k median input tokens while GPT reports 95k to 175k. That is not GPT reading five to nine times more of your code. The two CLIs count context resends and cache reads differently, so their input-token numbers are measuring different things under the same label. The practical rule: within-model comparisons are safe, because each CLI is consistent with itself, while cross-model input-token or dollar comparisons are indicative only, and we refuse to publish them as hard claims. The axes that are honestly comparable across models are output tokens, turns, and wall-clock time. Those are what we lean on.

Can You Trust an AI to Judge Another AI's Code?

Only with guardrails. We stripped identity from every transcript and had two separate models grade everything, because a single-model judge carries a documented self-preference bias. Our two judges agreed within one point on 96 percent of runs.

Running two judges was not decoration. Automated judges carry a documented self-preference bias: the paper Self-Preference Bias in LLM-as-a-Judge (Wataoka et al., 2024) shows a model systematically favoring certain outputs when it grades, apparently because it rewards text that is more familiar, lower-perplexity, to itself. A single-model judge grading its own model's work is a conflict of interest baked into the evaluator.

So we stripped identity from every transcript and had both models grade everything. Across 139 dual-judged runs, the two judges landed within one point of each other on 96 percent of runs, with a mean absolute gap of 0.53 points on a ten-point scale and a Pearson correlation of 0.76. On the pilot, the GPT judge ran about half a point more generous than the Claude judge on the same diffs, and both judges leaned slightly toward GPT's code. That last detail is exactly why the judging is blind and dual, and exactly why we report the disagreement instead of averaging it away.

What Can This Experiment Not Tell You?

It cannot rank the models, because both passed everything. It is a ceiling result on a suite whose difficulty was mis-calibrated low, judged by models rather than humans, on small per-arm samples. That mis-calibration is the headline finding, not a footnote.

This section is the credibility, so we are going to be blunt about what this experiment cannot tell you.

• The sample is small per arm. With 24 trials per arm, the design has 80 percent power to detect only large gaps, roughly 36.7 percentage points or more at a 50 percent baseline. A real difference smaller than that would not show up. It gets a confidence interval, not a verdict.
• It is a ceiling effect. Because both models passed everything, this suite cannot rank them. It can only establish that both clear this tier. Anyone who reads 154 perfect runs as the models are identical is over-reading it. The correct claim is narrower.
• The judges are models, not humans. Blind and dual and with disagreement reported, but still automated graders with their own biases.
• One run was re-run. A single Claude greenfield run hit a transient CLI error, 2.3 seconds and zero tokens, and was re-run. Disclosed.
• The tasks came from an AI-assisted pipeline. They were proven solvable, but their difficulty was mis-calibrated low. That miscalibration is not a footnote. It is the headline finding.
• Cost figures use list-price placeholders. We are not publishing dollar claims off vendor-incomparable token counts.
• One model may have operated in the dumb zone. Practitioners who run these models at scale, Matt Pocock loudest among them, observe that output quality degrades once context grows past roughly 100k tokens, a pattern consistent with published long-context research (Liu et al., 2024). By its own CLI's accounting, GPT-5.6 exceeded 100k cumulative input tokens on 65 of its 88 runs (median 142k, max 353k); Claude peaked at 19.6k. Two honest qualifiers: codex counts context resends across internal steps, so live context occupancy at any moment is not observable from our data, and the bias runs against GPT, which passed everything anyway. The tie is therefore safe, but the blind-judge quality comparisons carry this asymmetry, and it is on the record.

What Happens in the Next Experiment?

The next round raises the difficulty until failures appear. A model gap only becomes measurable once at least one model starts missing, and the effort dial only earns its cost once thinking harder changes an outcome instead of just a token count.

The most useful result of a ceiling is that it tells you exactly where to point the next experiment. Both models beat this tier, so the tier was too easy, so the tier goes up. Experiment 2 is the hard tier: the same rig, the same blocking, the same blind dual-judge, the same exact statistics, and harder tasks pushed up in difficulty until failures start appearing. Failures are what you need. A gap only becomes measurable once at least one model starts missing, and the effort dial only earns its cost once think harder changes an outcome instead of just a token count. Same exam room, harder exam. That is where Claude versus GPT gets a real answer, and we will report that one the same way we reported this one: with the intervals drawn in, the disagreements shown, and no winner invented that the data did not earn.