We Ran Kimi K3 Against the Frontier on Our Own Coding Gauntlet

Why Not Just Read the Leaderboard?

Because a leaderboard rank is a summary statistic over someone else's task distribution, not performance on yours. Public benchmarks suffer contamination and overfitting, and model outputs are probabilistic, so a single rank cannot tell you how a model behaves on your data under your constraints. You have to run your own eval.

Chip Huyen's case for evaluation-driven development starts from a property of foundation models that breaks classic testing: their outputs are probabilistic, not deterministic. The same prompt can produce different answers, so a binary pass or fail unit test stops being a clean signal, and you need evaluation that scores a distribution of behavior rather than asserting one correct string.

Her critique of leaderboards follows from there. Public benchmarks like MMLU are a bird's-eye view: useful for a first-pass sanity check, useless as a final selection criterion. They suffer contamination, where test items leak into training data, and overfitting, where teams tune toward the benchmark. Her prescription is to define what good means first, then run a funnel: a hard-constraint gate on license, hosting, and hardware, then public benchmarks for a cheap first cut, then your own private eval judged side-by-side on quality, cost, and latency. This report is that third step for K3. It is the same position we took in our report on AI agent evaluation: public benchmarks measure the model, not your system.

What Is Kimi K3?

Kimi K3 is an open Mixture-of-Experts model from Moonshot AI, released July 16, 2026, with 2.8 trillion parameters, a one-million-token context window, and reasoning fixed at max effort. Its list price is far below the American frontier: $3 per million input tokens and $15 per million output, roughly five times cheaper per token than the Opus and GPT tier.

We attach a confidence flag to every external claim, because not trusting the vendor's numbers is the entire point of the exercise. Confirmed from a live model probe and independent write-ups: K3 released on July 16, 2026, is a 2.8-trillion-parameter open Mixture-of-Experts, carries a 1,048,576-token (1M) context window, and runs reasoning at max effort only, so there is no effort ladder to sweep. Pricing is $0.30 per million cache-hit input tokens, $3.00 cache-miss, and $15.00 per million output tokens. Third-party trackers put its long-horizon Elo just behind Fable 5 at launch. We deliberately do not lean on Moonshot's own benchmark table.

How Did We Test It?

We ran nine hand-authored, contamination-resistant coding tasks: four seeded bugs, four ticket-sized features, and one greenfield CLI, in Python and TypeScript. We define done, not the model: a task's verify script must exit zero. Every passing diff was scored blind by two judges, Claude Opus 4.8 and GPT-5.6, on four quality axes.

The gauntlet is nine tasks we wrote ourselves, novel repos rather than public benchmark problems that could sit in a training set. Tier 1 is a seeded bug with a failing test (four tasks). Tier 2 is a ticket-sized feature whose tests reference functions that do not yet exist (four tasks). Tier 3 is a greenfield CLI built from scratch against a hidden acceptance suite (one task). Done is defined by us: we run the task's verify script and exit zero means pass. The model never grades its own work. Runs are interleaved on a fixed seed and executed one at a time with resumable state.

Quality on passing runs is scored by two judges, Claude Opus 4.8 and GPT-5.6, each rating the diff from 0 to 10 on four axes: correctness beyond the visible tests, simplicity and minimal diff, idiomatic style, and spec adherence, with no model identity in the judge prompt. Judging cross-vendor and blind neutralizes the self-grading objection we cover in the 154-trial companion.

One confound we disclose rather than hide: K3 has no native agent CLI that fits our stack, so we drove it through Claude Code pointed at Moonshot's Anthropic-compatible endpoint. The upside is that K3 and Fable then run inside the same agent scaffold, so the Fable-versus-K3 comparison isolates the model cleanly. The cost is that GPT-5.6-Sol runs inside Codex, a different scaffold. This is a study of how well each model does the job inside a capable agent, not a pure model-in-a-vacuum study. Fable-versus-K3 is clean; anything involving Sol carries the scaffold caveat.

What Did the Gauntlet Show?

All three models cleared the tasks at a near-ceiling pass rate, so pass rate was not the interesting variable. K3 passed 39 of 40 runs; its single miss was a transport error with zero tokens and a timeout, not a wrong answer. With everyone near 100 percent, the real variation lives in cost, quality, and latency.

Across 194 runs, 193 passed and were judged. All three models clear these tasks at a near-ceiling pass rate. K3 passed 39 of 40; its single miss was a CLI transport error, zero tokens and a wall-clock timeout, not a wrong answer, and its other repetitions of that task passed. Pass rate is therefore a near-constant here.

Which Model Is Cheapest to Run?

Kimi K3, by a wide margin. At list price it cost about $0.14 per task, against Fable's $0.44 and Sol's $1.62. The counterintuitive part is that K3 spends roughly twice Fable's tokens per solved task, yet costs about a third as much, because its per-token price is around five times lower. Token count is not the bill. Price times tokens is.

Cost is the headline. K3 is the cheapest model on test by a wide margin, at about $0.14 per task against Fable's $0.44 and Sol's $1.62, while its judged quality sits squarely between the other two. The result is counterintuitive on tokens alone: K3 spends roughly twice Fable's tokens per solved task (about 39k versus 21k), yet costs about a third as much, because Moonshot's per-token price is roughly five times below the Opus and GPT tier. This is the same routing discipline we cover in our report on token efficiency: the sticker you should watch is price times tokens, not tokens alone.

Does Kimi K3 Write Good Code?

Yes, and it lands in the middle of the pack. Broken out by judge axis, K3 is never worst and never best. It edges Fable on correctness and idiomatic style and trails Sol across the board, but all three cluster inside about 0.7 points. Since the two judges themselves agree only within about half a point, those gaps are directional, not decisive.

Broken out by judge axis, K3 slots between Fable and Sol on every dimension. It edges Fable on correctness (8.82 versus 8.74) and idiomatic style (8.73 versus 8.47), and trails Sol across the board, with Sol leading on spec adherence at 9.48. All three cluster inside about 0.7 points. Given that the two judges themselves agree only to within about 0.56 points on average, these axis gaps are directional, not decisive. K3 is a consistent middle: never worst, never best.

Is Kimi K3 Fast?

No. K3 was the slowest model on the clock, a median of 89 seconds per task against Fable's 64 and Sol's 54. That is expected: K3 runs at max effort only, with no lower setting to trade down for speed, and our harness capped its usable context at 200k rather than the advertised one million. Cheap and slow is the trade.

K3 is the slowest model on wall-clock, a median of 89 seconds per task against Fable's 64 and Sol's 54. That is expected. It is max-effort only, with no lower-effort setting to trade down, and Claude Code capped its usable context at 200k rather than the advertised 1M. If your workload is latency-sensitive, that is the cost of the low price.

What Is the Scaffold Tax?

It is the fixed token cost an agent harness adds to every model call before the model sees a line of your code. In our setup that preamble was about 29 to 30 thousand tokens. Asking K3 to reply with the single word OK cost over 29,000 input tokens. On short tasks that cover charge, not the model, is most of the bill.

The single most transferable finding has nothing to do with which model won. It is that per-task cost is a property of your harness as much as of the model's sticker price, and we can put a number on it. Every time you hand a coding model a task through an agent like Claude Code, the agent prepends a large fixed preamble before the model sees a single line of your code: the system prompt plus the full JSON schema for every tool the model may call. In our setup that preamble alone was about 29 to 30 thousand tokens. We measured it directly: asking K3 to reply with the single word OK cost 29,091 input tokens.

Think of it as a cover charge at the door. You pay it to get in, whether you order a one-line bug fix or a from-scratch CLI. Nearly every real K3 call landed at 30 to 43k input tokens, stacked on top of that fixed floor, no matter how little the model wrote back. Two consequences follow. On short tasks, that fixed preamble is most of the input bill, and the model's efficiency barely matters, the harness does. And this is exactly the quantity a leaderboard cannot show you, because a leaderboard never runs the model inside your real agent with your real tools. If you are selecting a model to deploy, measure it inside the agent you will ship.

How Solid Are These Numbers?

Every test is exact, not a large-sample approximation, and reported with raw counts. The cost gap between models is statistically real. The two judges are trustworthy, agreeing within one point on 94 percent of runs. And we are blunt about power: with roughly 24 runs per arm, the design can only detect large pass-rate gaps, so we say no detectable difference, never that the models are equal.

Every test is exact, rendered with raw counts rather than a bare p-value.

• The cost gap is real. A sign-flip permutation test on per-task median log-tokens, paired by task, has all nine tasks favoring the cheaper model, two-sided exact p = 0.0039. K3 sits below both on list-price cost per task.
• The judges are trustworthy. Over 178 dual-judged runs, the Claude and GPT judges correlate at Pearson r = 0.69 with a mean absolute gap of 0.56 points on the 0 to 10 scale, and 94 percent of runs agree within one point.
• We are honest about power. With about 24 runs per arm, the design detects only pass-rate gaps of roughly 37 percentage points at 80 percent power. Every model here passes near 100 percent, so we say no detectable pass-rate difference, never that the models are equal.

What Can This Study Not Tell You?

It cannot rank the models on quality, because all three passed nearly everything. It carries a scaffold confound, since Sol ran in a different agent than K3 and Fable. K3 is max-only, the samples are small, and list-price dollars are estimates. Contamination can never be fully disproven. These are the limits, stated plainly.

• Scaffold confound. Fable and K3 run in Claude Code; Sol runs in Codex. Cross-scaffold comparisons are representative of real deployment, not pure model isolation. Fable-versus-K3 is clean; anything involving Sol carries the caveat.
• K3 is max-only. There is no effort ladder to sweep, so its latency and cost reflect its single available setting.
• Small samples, especially on the single Tier 3 task.
• Effort mix. The pooled cost figure averages over each model's available efforts, which differ. The K3-cheapest and Sol-most-expensive ordering is robust, but the exact Fable-to-Sol ratio shifts with the effort mix you assume.
• List-price dollars are estimates. Subscription models are billed differently; only K3 is billed for real.
• Contamination cannot be fully disproven, though the tasks are novel repos we authored.
• Judge bias. The two judges are frontier models with their own biases, even blinded.

So Should You Use Kimi K3?

It depends on what you optimize for. On our tasks K3 is strong and cheap: it solves what the frontier solves, at judged quality between Fable 5 and Sol, for the lowest list-price cost of the three. If you can tolerate the slowest latency, it is a serious option. The bigger lesson is to measure any model inside the agent you will actually ship.

On our nine contamination-resistant coding tasks, Kimi K3 is strong and cheap. It solves what the frontier solves, at judged quality between Fable 5 and GPT-5.6-Sol, for the lowest list-price cost per task of the three, because its per-token price more than offsets its heavier token appetite. Whether it is the right choice for a given deployment depends on latency tolerance, since it is the slowest, on the confounds above, and on the most portable lesson here: the harness you run a model inside can move its per-task cost more than the choice of model does. Measure the model in the agent you will ship, define good yourself, and treat the leaderboard as a first cut rather than a verdict.