We Ran 154 Controlled Trials of Claude vs GPT. Both Scored Perfect.

Why Do Most Claude Versus GPT Comparisons Tell You Nothing?

Most comparisons run each model once, and one run tells you little because model output is non-deterministic. Ask the same model the same question twice and you can get two different answers. A fair comparison needs repeated runs under identical conditions, plus statistics that show how much uncertainty is left.

The underlying problem is non-determinism. Ask the same model the same coding question twice and the answer, the token count and the outcome can all change. A single run tells you what happened that once and nothing about what happens on average, which is what matters when you are deciding which tool your team ships with. Run the same comparison again tomorrow and the winner can flip.

Public benchmarks have a separate problem. The ones that do use repeated, structured evaluation are running out of headroom: frontier models now cluster near the top of the standard suites, which makes those suites worse at telling models apart. Contamination adds to it, because when benchmark problems, or text closely derived from them, land in a model's training data, the model can recall an answer instead of reasoning to it. The team behind LiveCodeBench (Jain et al., 2024) documented this for coding, showed that older suites are no longer sufficient on their own, and built a time-segmented benchmark of fresh contest problems to avoid the leakage. Our report on AI agent evaluation makes the same point: a public benchmark measures the model rather than the system you build around it.

How Do You Design a Fair Test of Two AI Coding Models?

We ran a full-factorial designed experiment blocked by task: model by reasoning effort by harness, every configuration facing every one of nine tasks, each task proven solvable before any model touched it. Runs were seed-interleaved and every outcome scored by a deterministic pass or fail gate.

We tested Claude Fable 5 through Anthropic's claude CLI and GPT-5.6 through OpenAI's codex CLI. The design is a full-factorial experiment blocked by task, with three factors: model (Claude, GPT), reasoning effort (two to three levels), and harness (a bare CLI versus a shared agent-instruction harness). Blocking means every configuration sees every task, so task difficulty cancels out of any paired comparison. Replication is n=3 repetitions per cell on the smaller tasks and n=2 on the greenfield build, because repeated runs let us treat each outcome as a Bernoulli trial and estimate a real pass rate. Run order is seed-interleaved (seed 1337) so that time-of-day and API-load drift cannot systematically favor one model.

There are nine tasks, and every one was proven solvable before any model touched it. Four are seeded-bug tasks with deliberately subtle defects: an off-by-one in pagination, a naive-versus-timezone-aware datetime bug, a cache key that silently drops a dimension, and an async dedup race. Four are feature tickets written in a strict six-section format. The ninth is a greenfield CLI build, a roughly 200-line cost-splitting tool with a reference implementation. Each task ships a self-test that fails on the broken starting code and passes after the reference patch. All nine were green before the experiment began, so any failure would have been the model's and could not be blamed on an impossible task.

Conditions were held byte-identical, with the same harness instruction files, the same prompts and the same deterministic verify gate deciding pass or fail. Every transcript and diff was then stripped of identifying markers and scored blind by two separate judges, one Claude and one GPT, on a four-dimension rubric. We used two judges because a single-model judge is a known failure mode, which we come back to below. The per-cell samples are small, so we used exact statistics instead of large-sample approximations: Wilson score intervals for every pass rate, McNemar's exact test for paired head-to-heads, an exact sign-flip permutation test on tokens, and an explicit power statement for what the sample can detect.

What Did 154 Trials Actually Show?

Both models passed everything solvable. Sweep 1 was 120 of 120; every solo configuration across both sweeps was 152 of 152. The paired McNemar test found zero discordant pairs, so there was no detectable difference between the models on this suite.

Sweep 1 was 120 passes out of 120 attempts, across both models and every effort level. Adding the solo runs from the second sweep (the harnessed tasks and the bare-plus-harnessed greenfield build) brings the total to 152 of 152 clean passes on every configuration where a model worked alone. A perfect score should be reported with its confidence interval rather than as a bare 100 percent. Twenty-four passes out of twenty-four gives a 95 percent Wilson interval of 86.2 percent to 100 percent. So we did not prove that the model never fails on tasks like these. We proved that if it has a failure rate, the data are consistent with anything from zero up to about 14 percent, and the chart at the top of this report draws every perfect bar with that uncertainty.

For the head-to-head we used McNemar's exact test, which only extracts information from discordant pairs: cases where one model passed and the other failed on the same task and repetition. There were zero discordant pairs, since every matched pair was a pass for both models, so there was nothing to test. What the data support is that there is no detectable difference between the models on this suite. That is a weaker claim than saying they are equal, and it is the only one we are making.

One configuration was inconsistent. We also tested a hybrid in which Claude orchestrated and GPT implemented. Across four attempts, two produced planning notes and zero code, and two shipped working solutions of roughly 160 to 176 lines, so the hybrid passed 2 of 4. The rig was identical across all four, so the most defensible reading is run-to-run variance in a multi-agent handoff rather than a fixed defect. With n=4 we can't say more than that, and we are carrying it into the next experiment as something to test.

Does Higher Reasoning Effort Improve Code Quality?

No. Higher reasoning effort did not improve results on this suite. Pass rate stayed at 100 percent at every effort level for both models, and only cost changed. GPT-5.6 at high effort used about 80 percent more output tokens than at low effort for the same result.

Both models expose a reasoning-effort setting, sold on the idea that higher effort makes the model think harder and do better. On this suite it made no difference to pass rate, which was 100 percent at every effort level for both models, but it did change cost. Claude at high effort spent about 16 percent more output tokens than at medium (2,008 versus 1,736 median) for the same result. GPT-5.6 at high effort spent about 80 percent more output tokens than at low (1,966 versus 1,095) and pulled roughly 1.8 times the input tokens (174.7k versus 95.2k). The pairwise McNemar tests between adjacent effort rungs again found zero discordant pairs. Our report on token efficiency argues for the same routing rule: use the expensive setting only where it changes an outcome.

The pattern matches published work on what researchers call the overthinking phenomenon. The survey Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (Sui et al., 2025) catalogs how reasoning models generate verbose, redundant chains that add computational cost without adding correctness. The empirical study Between Underthinking and Overthinking (Su et al., 2025) goes further and finds that models overthink easy problems, producing unnecessarily long outputs, because they are poor at gauging problem difficulty. In hindsight our nine tasks were easy for these models, so the extra effort paid for longer deliberation over answers the models already had.

Do Claude and GPT Solve Problems the Same Way?

No. Claude works over six to seven turns, reading, editing and checking until it converges, while GPT one-shots the task in a single turn. Both styles reached a passing solution on every task, so neither one won on outcomes in our 154-run test.

The traces show two different working styles. Claude iterates, with a median of 6 to 7 turns per task and roughly 59 to 68 seconds of wall-clock time: it reads, edits, checks and edits again until it converges on a fix. GPT one-shots, with a median of a single turn and roughly 26 to 64 seconds, reasoning internally and emitting a solution in one pass. Neither style won on outcomes at this tier, because both cleared it. If you have watched these tools work and formed an opinion about which one works harder, keep in mind that turn count describes a workflow and tells you nothing by itself about quality.

Input tokens cannot be compared across vendors. The config table shows Claude reporting around 19.4k median input tokens while GPT reports 95k to 175k, but that gap does not mean GPT read five to nine times more of your code. The two CLIs count context resends and cache reads differently, so their input-token numbers measure different things under the same label. Within-model comparisons are safe, because each CLI is consistent with itself, while cross-model input-token or dollar comparisons are indicative only, and we don't publish them as hard claims. The measures that do compare across models are output tokens, turns and wall-clock time, and those are the ones we rely on.

Can You Trust an AI to Judge Another AI's Code?

Only with safeguards against bias. We stripped identity from every transcript and had two separate models grade everything, because a single-model judge carries a documented self-preference bias. Our two judges agreed within one point on 96 percent of the 139 dual-judged runs.

We ran two judges because automated judges carry a documented self-preference bias. The paper Self-Preference Bias in LLM-as-a-Judge (Wataoka et al., 2024) shows a model systematically favoring certain outputs when it grades, apparently because it rewards text that is more familiar to it (lower perplexity). When one model judges work produced by the same model, the evaluator has a conflict of interest built in.

We stripped identity from every transcript and had both models grade everything. Across 139 dual-judged runs, the two judges landed within one point of each other on 96 percent of runs, with a mean absolute gap of 0.53 points on a ten-point scale and a Pearson correlation of 0.76. On the pilot, the GPT judge scored about half a point more generously than the Claude judge on the same diffs, and both judges leaned slightly toward GPT's code. That lean is why the judging is blind and dual, and why we report the disagreement instead of averaging it away.

What Can This Experiment Not Tell You?

It cannot rank the models, because both passed everything. It is a ceiling result on a suite whose difficulty was mis-calibrated low, judged by models rather than humans, with small samples per arm. That mis-calibration is the main finding, and the next suite will be harder.

These are the limits of the experiment and what each one means for the results.

• The sample is small per arm. With 24 trials per arm, the design has 80 percent power to detect only large gaps, roughly 36.7 percentage points or more at a 50 percent baseline. A real difference smaller than that would not show up, so all we can report for it is a confidence interval.
• It is a ceiling effect. Because both models passed everything, this suite cannot rank them and can only establish that both clear this tier. Reading 154 perfect runs as proof that the models are identical over-reads the result.
• The judges are models. The judging was blind and dual and we report disagreement, but they are still automated graders with their own biases.
• One run was repeated. A single Claude greenfield run hit a transient CLI error (2.3 seconds, zero tokens) and was re-run.
• The tasks came from an AI-assisted pipeline. They were proven solvable, but their difficulty was mis-calibrated low, and that is the main finding of this experiment.
• Cost figures use list-price placeholders. We don't publish dollar claims built on token counts the vendors measure differently.
• One model may have operated in the dumb zone. Practitioners who run these models at scale, Matt Pocock most prominently, observe that output quality degrades once context grows past roughly 100k tokens, a pattern consistent with published long-context research (Liu et al., 2024). By its own CLI's accounting, GPT-5.6 exceeded 100k cumulative input tokens on 65 of its 88 runs (median 142k, max 353k), while Claude peaked at 19.6k. Two qualifiers apply. Codex counts context resends across internal steps, so live context occupancy at any moment is not observable from our data, and the bias runs against GPT, which passed everything anyway. The tie therefore stands, but the blind-judge quality comparisons carry this asymmetry, and we are recording it here.

What Happens in the Next Experiment?

The next experiment raises task difficulty until failures appear. A gap between the models only becomes measurable once at least one of them starts missing, and higher reasoning effort is only worth paying for once it changes an outcome and not just the token count.

A ceiling result is still useful, because it tells us where to aim the next experiment. Both models beat this tier, so Experiment 2 moves to a hard tier with the same rig, the same blocking, the same blind dual-judge and the same exact statistics, and tasks pushed up in difficulty until failures start appearing. We need those failures, because a gap between the models only becomes measurable once at least one of them starts missing, and the effort setting only justifies its cost once thinking harder changes an outcome. That experiment is where Claude versus GPT gets a real answer, and we will report it the same way as this one, with the intervals drawn in, the judges' disagreements shown, and no winner named unless the data support one.