We Went Looking for Context Rot at 700,000 Tokens

What is context rot, exactly?

Context rot is recall degrading as the context window fills, measured on a task whose difficulty is held constant. It is a different problem from the model running out of room, and from an instruction file growing too long. The mechanism behind it is attention spreading thinner across more tokens.

People use the term in two senses. One describes instruction files bloating until the important rules sit in the middle, where models attend worst, which is a document design problem with a document design fix. The other is narrower, and it is the sense that Chroma's report and the NoLiMa paper measure and that this study tests: keep the task identical, add filler around the answer, and measure how far accuracy falls. Our note on context engineering uses the first sense, and this page uses the second.

The proposed mechanism follows from how attention is computed. Attention is a softmax over every token in the window, and the mass sums to one no matter how many tokens there are. Its entropy grows roughly with the log of the count, so attention flattens toward uniform as context grows. An exact string match still produces a spike sharp enough to survive that flattening, but a weak semantic association does not.

Why did our first two attempts find nothing?

Both early versions planted a key and then asked for it by name. Exact string matching produces a sharp attention spike that survives dilution, so the model scored 1,251 out of 1,251 keys correct through 700,000 tokens. That result measured the one retrieval mode immune to the mechanism we were hunting.

A perfect score from that design was no evidence that long context is fine, only that the instrument was pointed at the wrong thing. Version two returned 100 percent at every length and every depth decile, with zero decoy grabs, which is exactly what the softmax argument predicts for literal matching. The null result was correct, but it could not tell us anything about context rot.

At that point the risk was that we would keep tuning until something broke and then publish the break. So we wrote the version three design down first, in a pre-registration file committed before any version three run, with the thresholds copied from published work rather than chosen after seeing our own numbers. Two of those thresholds mattered most. The first required difficulty to calibrate into a 90 to 95 percent band at the short control length, because at 100 percent a null is indistinguishable from a saturated instrument. The second defined degradation as the short-to-long gap on an identical task, never an absolute score, so a harder question could not pass for context rot.

How do you build a probe that context rot can actually fail?

Remove every lexical handle the model could match on. One arm states a platform ceiling in one half of the window and this service's fraction in the other, and never writes the number anywhere. The other arm plants two fields per service in separate files and asks a question that filters on one and selects on the other.

Both arms use the same layout. The prompt is a long session log of files supposedly read earlier, with the planted material scattered at controlled depths, and the reply has to follow a strict one-line-per-answer output contract so that grading never depends on parsing prose.

• The latent arm asks for values like timeout_ms and queue_depth, and the number appears nowhere in the window in any unit. A platform page in the shallow half states a base quantity in words, such as a call chain being given one full second. A runbook page in the deep half states this service's fraction of it, so neither block alone yields an answer.
• The fractions are trivial on purpose (halves, quarters and thirds), because the thing under test is holding two separated facts together and arithmetic should not be a factor. An import-time assertion checks that every relation is a proper fraction of its anchor, so the two tables cannot drift apart silently.
• Semantic decoys state a superseded value in equally plausible prose, marked as an earlier revision that was later replaced. Discarding one requires applying the recency rule in the brief rather than doing a second lookup.
• The join arm plants retry_budget for each service in one file and timeout_ms in a different file far away, then asks which service has the highest timeout_ms among those passing a retry_budget filter. Draws are rejected unless the answer differs from the globally highest timeout_ms, so a model that skips the shallow half and always names the global maximum scores badly.
• Question order is shuffled away from depth order, so reading the log in sequence gives the model no advantage.

What did 128 runs at up to 730,000 tokens show?

The latent arm held at 100 percent through 34,000 tokens and 99.3 percent at 80,000, then dropped to 94.4 percent at 730,000. The join arm held at 99.2 percent at both lengths it ran. Against NoLiMa's 85 percent retention bar, effective length is at least 700,000 tokens.

We computed the intervals on instances rather than on keys, which costs a lot of apparent precision. The nine keys inside one run share a prompt and a haystack, so they are clustered observations rather than independent ones, and version two's 180 key-observations per length were effectively 20. Counting instances is why a 270 out of 270 result still carries a lower bound of 88.6 percent.

For comparison, NoLiMa tested twelve models on a similar latent-association task, and ten of them had fallen below half their short-context baseline by 32,000 tokens. GPT-4.1 posted an effective length of 16,000 against a claimed one million. No Claude model appeared in those tables, and nothing in that paper ran past 128,000. A year later, on our version of that task, retention at 730,000 tokens is 94.4 percent.

Why is the smallest context in this study 34,000 tokens?

The smallest context is 34,000 tokens because the Claude Code context window is never empty. The CLI sends a system prompt and a full set of tool definitions with every request, and that overhead measured a steady 29,000 tokens across all 128 runs. Even our nominal 5,000 token cell sits past the point where most NoLiMa models had collapsed.

We asked for 5,000 tokens of context and measured 33,766, and we asked for 50,000 and measured 80,132. The gap is flat, which is what lets us identify it as fixed harness overhead rather than anything about the task. It also means our control length is not a short-context baseline in the NoLiMa sense at all, since it sits past their 32,000 token column, where most of their models had already lost half their accuracy.

In practice, if you run agents through a CLI or an SDK with tools attached, your usable window is smaller than the advertised number, and every request spends that overhead before your content arrives. Measure it instead of assuming it, and count it in any token budget you write down.

What does a failure at 700,000 tokens actually look like?

Every one of the six latent misses returned the platform base quantity without applying the service fraction, so the model found the shallow half of the pair and went no further. None of the misses across 128 runs picked a superseded decoy value, and no join question ignored its filter.

This failure is specific and fairly encouraging. The model avoids the stale values and keeps track of where the answer is, but after retrieving the first fact it does not carry it over to the second. The four misses at 730,000 sit at planted depths of 0.55 to 0.75, inside the same range as the correct answers, so with four data points there is no depth signal to read.

One bookkeeping detail changed the scorer. The two misses at 80,000 tokens were originally filed as "other" because no category existed for this failure yet, even though both had returned 1,024 for queue_depth, the untouched platform anchor. With no category of its own, the failure mode looked as though it had never occurred, so we added the category. An assertion at import now checks the anchor prose against the anchor numbers, because a desync there would make the new label silently stop matching instead of crashing.

Does this refute the 100,000 token dumb zone?

No, because this study tests a different claim. Matt Pocock's dumb zone describes a working session degrading as it accumulates its own noise, judged on instruction following and decision quality. Our study holds one task fixed, fills the window with inert filler, and scores recall and composition, so it measures a different failure mode.

Pocock's own dictionary states the claim more carefully than the repetitions of it do: early in a session the agent is in a "smart zone", sharp and focused, and as the session grows it drifts into a dumb zone, sloppier, forgetful, more mistakes, with the same model and harness and only more context. It puts the onset around 125K to 150K tokens and adds that this is debated, a hedge worth keeping. Writing up his own runs against a one million token window, he was blunter: "Don't treat 1M context window any differently. It's still 100K of smart, and 900K of dumb." He credits the phrase itself to Dex Horthy.

A working session contains three things our prompt does not: tool results nobody cleared, failed attempts and abandoned plans nobody swept up, and the model's own earlier output fed back in as input. Our haystack is inert filler, written once and never touched, with a fixed question asked cold at the end. That gap is the whole difference between the two experiments, and it is wider than the matching token counts make it look. His own illustration is a session rather than a prompt: an agent stuck on one problem for an hour somewhere around 150K to 200K tokens, which solved it quickly once the context was cleared. Our design cannot produce that outcome because nothing in it accumulates.

The two results measure different things, so they do not contradict each other. They overlap on recall, since the smart zone is described partly as recall being good early, and our data says recall survives to 730,000 tokens in a static prompt. They do not overlap on most of what the claim is actually about, because we never scored instruction adherence and never let a session generate its own noise. Our only failures landed one step past plain retrieval, where two separated facts have to be held together, which suggests that composition breaks first.

We have not run the experiment that would test the dumb zone directly. It would hold a task fixed, grow a real session around it with actual tool output and actual dead ends, and score instruction adherence instead of recall. Until someone runs it, this post is evidence about long prompts and says nothing about long sessions.

What should you change about how you use long context?

Less than the warnings about context rot suggest and more than long-context marketing implies, judged on recall and composition in a fixed task. Recall out to 700,000 tokens held up well enough that compacting by reflex probably costs more than it saves, but composition across distant facts degrades first. None of this covers a long working session.

• Measure your own floor before you trust a window size. Harness overhead is a fixed amount, and you can read it off a usage field.
• Put facts that have to be combined near each other. The one failure mode we caught is the cross-window join, so co-locating the pieces of an answer removes exactly the thing that broke.
• Treat quadratic attention cost as a separate argument from quality. Most of the case for keeping sessions under 100,000 tokens is about cost even when it is framed as quality, and the two should be priced separately.
• Do not read this as a general verdict, since it covers one model and two task shapes run through one CLI. Version two proved how easily a well-run experiment can measure the wrong thing and still score 100 percent.

We set out to find context rot, built a probe specifically designed to expose it, spent $83.36 finding out, and measured a 5.6 point drop at the far end of a window most tooling will not let you fill. The pre-registration says both outcomes were publishable before we knew which one we had. The runner, the pre-registration and every result row are in our internal repo, and the eight-run cell at 700,000 tokens is the first thing we would spend more on.