We Ran Kimi K3 Against the Frontier on Our Own Coding Gauntlet

Why Not Just Read the Leaderboard?

A leaderboard rank is a summary statistic over someone else's task distribution and says little about performance on your tasks. Public benchmarks also suffer contamination and overfitting, and model outputs are probabilistic, so a single rank cannot tell you how a model behaves on your data under your constraints. You need your own eval for that.

Chip Huyen's case for evaluation-driven development starts from a property of foundation models that breaks classic testing, which is that their outputs are probabilistic. The same prompt can produce different answers, so a binary pass or fail unit test stops being a clean signal, and you need evaluation that scores a distribution of behavior rather than asserting one correct string.

Her critique of leaderboards follows from that. She treats public benchmarks like MMLU as a bird's-eye view that is useful for a first-pass sanity check but no use as a final selection criterion. They suffer contamination, where test items leak into training data, and overfitting, where teams tune toward the benchmark. Her prescription is to define what good means first and then run a funnel: a hard-constraint gate on license, hosting and hardware, then public benchmarks for a cheap first cut, then your own private eval judged side-by-side on quality, cost and latency. This report is that third step for K3. Our report on AI agent evaluation takes the same position, that a public benchmark measures the model rather than the system you build around it.

What Is Kimi K3?

Kimi K3 is an open Mixture-of-Experts model from Moonshot AI, released July 16, 2026, with 2.8 trillion parameters, a one-million-token context window, and reasoning fixed at max effort. Its list price is far below the American frontier: $3 per million input tokens and $15 per million output, roughly five times cheaper per token than the Opus and GPT tier.

We attach a confidence flag to every external claim, since the point of the exercise is to avoid relying on the vendor's numbers. These facts are confirmed from a live model probe and independent write-ups: K3 was released on July 16, 2026, is a 2.8-trillion-parameter open Mixture-of-Experts, has a 1,048,576-token (1M) context window, and runs reasoning at max effort only, so there is no effort ladder to sweep. Pricing is $0.30 per million cache-hit input tokens, $3.00 cache-miss, and $15.00 per million output tokens. Third-party trackers put its long-horizon Elo just behind Fable 5 at launch. We deliberately do not rely on Moonshot's own benchmark table.

How Did We Test It?

We ran nine hand-authored, contamination-resistant coding tasks in Python and TypeScript: four seeded bugs, four ticket-sized features and one greenfield CLI. A task counts as done only when its verify script exits zero, and we run that script ourselves. Every passing diff was scored blind by two judges, Claude Opus 4.8 and GPT-5.6, on four quality axes.

The gauntlet is nine tasks we wrote ourselves in novel repos, so none of them are public benchmark problems that could sit in a training set. Tier 1 is a seeded bug with a failing test (four tasks). Tier 2 is a ticket-sized feature whose tests reference functions that do not yet exist (four tasks). Tier 3 is a greenfield CLI built from scratch against a hidden acceptance suite (one task). We run each task's verify script ourselves and count an exit code of zero as a pass, so the model never grades its own work. Runs are interleaved on a fixed seed and executed one at a time with resumable state.

Quality on passing runs is scored by two judges, Claude Opus 4.8 and GPT-5.6, each rating the diff from 0 to 10 on four axes: correctness beyond the visible tests, simplicity and minimal diff, idiomatic style, and spec adherence, with no model identity in the judge prompt. Judging blind and across vendors neutralizes the self-grading objection we cover in the 154-trial companion.

There is one confound to disclose. K3 has no native agent CLI that fits our stack, so we drove it through Claude Code pointed at Moonshot's Anthropic-compatible endpoint. That puts K3 and Fable inside the same agent scaffold, so the Fable-versus-K3 comparison isolates the model cleanly, but GPT-5.6-Sol runs inside Codex, which is a different scaffold. What we measured is how well each model does the job inside a capable agent, which is different from testing the model in isolation. Comparisons between Fable and K3 are clean, and anything involving Sol carries the scaffold caveat.

What Did the Gauntlet Show?

All three models cleared the tasks at a near-ceiling pass rate, so pass rate did little to separate them. K3 passed 39 of 40 runs, and its one miss was a transport error that timed out with zero tokens returned. With every model near 100 percent, the differences show up in cost, quality and latency.

Across 194 runs, 193 passed and were judged, so all three models clear these tasks at a near-ceiling pass rate. K3 passed 39 of 40. Its single miss was a CLI transport error with zero tokens and a wall-clock timeout rather than a wrong answer, and its other repetitions of that task passed. Pass rate is therefore close to constant here.

Which Model Is Cheapest to Run?

Kimi K3 was the cheapest by a wide margin. At list price it cost about $0.14 per task, against Fable's $0.44 and Sol's $1.62. K3 spends roughly twice Fable's tokens per solved task yet costs about a third as much, because its per-token price is around five times lower.

K3 is the cheapest model on test by a wide margin, at about $0.14 per task against Fable's $0.44 and Sol's $1.62, while its judged quality sits between the other two. Token counts alone point the other way: K3 spends roughly twice Fable's tokens per solved task (about 39k versus 21k), yet costs about a third as much, because Moonshot's per-token price is roughly five times below the Opus and GPT tier. Our report on token efficiency makes the same routing point, which is to compare models on price multiplied by tokens, because token counts alone can mislead.

Does Kimi K3 Write Good Code?

Yes. Its judged quality sits between the other two models on every judge axis. It edges Fable on correctness and idiomatic style and trails Sol across the board, but all three cluster inside about 0.7 points, and the two judges themselves agree only within 0.56 points, so these gaps show a direction but are too small to settle a ranking.

Broken out by judge axis, K3 slots between Fable and Sol on every dimension. It edges Fable on correctness (8.82 versus 8.74) and idiomatic style (8.73 versus 8.47), and trails Sol across the board, with Sol leading on spec adherence at 9.48. All three cluster inside about 0.7 points, and since the two judges themselves agree only to within about 0.56 points on average, these axis gaps show a direction but are too small to settle a ranking.

Is Kimi K3 Fast?

No. K3 was the slowest model, with a median of 89 seconds per task against Fable's 64 and Sol's 54. We expected that, because K3 runs at max effort only with no lower setting to trade down for speed, and our harness capped its usable context at 200k instead of the advertised one million.

K3 is the slowest model on wall-clock, with a median of 89 seconds per task against Fable's 64 and Sol's 54. That is expected, because K3 is max-effort only, with no lower-effort setting to trade down to, and Claude Code capped its usable context at 200k of the advertised 1M. If your workload is latency-sensitive, the extra time is what you pay for the low price.

What Is the Scaffold Tax?

The scaffold tax is the fixed token cost an agent harness adds to every model call before the model sees your code. In our setup that preamble was about 29 to 30 thousand tokens, and asking K3 to reply with the single word OK cost over 29,000 input tokens. On short tasks this fixed cost is most of the bill.

The result most likely to apply to other setups is that per-task cost depends on your harness as much as on the model's sticker price, and we can put a number on it. Every time you hand a coding model a task through an agent like Claude Code, the agent prepends a large fixed preamble before the model sees a single line of your code, made up of the system prompt plus the full JSON schema for every tool the model may call. In our setup that preamble alone was about 29 to 30 thousand tokens. We measured it directly by asking K3 to reply with the single word OK, which cost 29,091 input tokens.

The preamble works like a cover charge: you pay it whether the task is a one-line bug fix or a from-scratch CLI. Nearly every real K3 call landed at 30 to 43k input tokens, built on that fixed floor, no matter how little the model wrote back. On short tasks the fixed preamble is most of the input bill, so the harness matters more to cost than the model's efficiency does. A leaderboard cannot show you this number, because it never runs the model inside your real agent with your real tools. If you are selecting a model to deploy, measure it inside the agent you will ship.

How Solid Are These Numbers?

Every test is exact and reported with raw counts. The cost gap between models is real, and the two judges agree within one point on 94 percent of runs. With roughly 24 runs per arm, though, the design detects only large pass-rate gaps, so we report no detectable difference and never claim the models are equal.

Every test is exact, rendered with raw counts rather than a bare p-value.

• The cost gap is real. A sign-flip permutation test on per-task median log-tokens, paired by task, has all nine tasks favoring the cheaper model, two-sided exact p = 0.0039. K3 sits below both on list-price cost per task.
• The judges are trustworthy. Over 178 dual-judged runs, the Claude and GPT judges correlate at Pearson r = 0.69 with a mean absolute gap of 0.56 points on the 0 to 10 scale, and 94 percent of runs agree within one point.
• Statistical power is limited. With about 24 runs per arm, the design detects only pass-rate gaps of roughly 37 percentage points at 80 percent power. Every model here passes near 100 percent, so we report no detectable pass-rate difference and do not claim the models are equal.

What Can This Study Not Tell You?

It cannot rank the models on quality, because all three passed nearly everything. It also carries a scaffold confound, since Sol ran in a different agent than K3 and Fable. K3 is max-only, the samples are small, list-price dollars are estimates, and contamination can never be fully disproven.

• Scaffold confound. Fable and K3 run in Claude Code, and Sol runs in Codex. Comparisons across scaffolds reflect real deployment but do not isolate the model. Fable-versus-K3 is clean, and anything involving Sol carries the caveat.
• K3 is max-only. There is no effort ladder to sweep, so its latency and cost reflect its single available setting.
• Small samples, especially on the single Tier 3 task.
• Effort mix. The pooled cost figure averages over each model's available efforts, which differ. The ordering (K3 cheapest, Sol most expensive) is stable, but the exact Fable-to-Sol ratio shifts with the effort mix you assume.
• List-price dollars are estimates. Subscription models are billed differently; only K3 is billed for real.
• Contamination cannot be fully disproven, though the tasks are novel repos we authored.
• Judge bias. The two judges are frontier models with their own biases, even blinded.

So Should You Use Kimi K3?

It depends on what you optimize for. On our tasks K3 solved what the frontier models solved, at judged quality between Fable 5 and Sol and the lowest list-price cost of the three. If you can tolerate the slowest latency of the group, it is a serious option. Whichever model you pick, measure it inside the agent you will ship.

On our nine contamination-resistant coding tasks, Kimi K3 solved what the frontier models solved, at judged quality between Fable 5 and GPT-5.6-Sol, for the lowest list-price cost per task of the three, because its per-token price more than offsets its heavier token use. Whether it is the right choice for a given deployment depends on how much latency you can accept, since it is the slowest, and on the confounds above. It also depends on the harness, because the harness you run a model inside can move its per-task cost more than the choice of model does. Measure the model in the agent you will ship, decide what good means for your work, and use the leaderboard only as a first cut.