We Went Looking for Context Rot at 700,000 Tokens

What is context rot, exactly?

Context rot is recall degrading as the context window fills, measured on a task whose difficulty is held constant. It is not the model running out of room, and it is not your instruction file being too long. The mechanism under it is attention spreading thinner across more tokens.

The term gets used two ways and the difference matters. One usage describes instruction files bloating until the important rules sit in the middle where models attend worst, which is a document design problem with a document design fix. The other, which is the one Chroma's report and the NoLiMa paper measure and the one this study tests, is narrower: hold the task identical, add filler around the answer, and watch accuracy fall. Our note on context engineering uses the first sense. This page uses the second.

The proposed mechanism is unglamorous. Attention is a softmax over every token in the window, and the mass sums to one no matter how many tokens there are. Its entropy grows roughly with the log of the count, so attention flattens toward uniform as context grows. An exact string match still produces a sharp spike that survives that flattening. A weak semantic association does not.

Why did our first two attempts find nothing?

Both early versions planted a key and then asked for it by name. Exact string matching produces a sharp attention spike that survives dilution, so the model scored 1,251 out of 1,251 keys correct through 700,000 tokens. That result measured the one retrieval mode immune to the mechanism we were hunting.

A perfect score is not evidence that long context is fine. It is evidence that the instrument was pointed at the wrong thing. Version two returned 100 percent at every length, every depth decile, and zero decoy grabs, which is exactly what the softmax argument predicts for literal matching. The null was correct and uninformative at the same time.

The risk at that point is obvious: keep tuning until something finally breaks, then publish the break. So we wrote the version three design down first, in a pre-registration file committed before any version three run, with the thresholds copied from published work rather than chosen after seeing our own numbers. Two of them did the heavy lifting. Difficulty had to calibrate into a 90 to 95 percent band at the short control length, because at 100 percent a null is indistinguishable from a saturated instrument. And degradation was defined as the short-to-long gap on an identical task, never an absolute score, so a harder question could not masquerade as context rot.

How do you build a probe that context rot can actually fail?

Remove every lexical handle. One arm states a platform ceiling in one half of the window and this service's fraction of it in the other half, never writing the number anywhere. The other arm plants two fields per service in separate files and asks a question that filters on one and selects on the other.

Both arms share a shape. The prompt is a long session log of files supposedly read earlier, the planted material is scattered at controlled depths, and the reply is bound to a strict one-line-per-answer output contract so grading never depends on parsing prose.

• The latent arm asks for values like timeout_ms and queue_depth, and the number appears nowhere in the window in any unit. A platform page in the shallow half states a base quantity in words, such as a call chain being given one full second. A runbook page in the deep half states this service's fraction of it. Neither block alone yields an answer.
• Fractions stay trivial on purpose, halves and quarters and thirds. The thing under test is holding two separated facts together, not arithmetic. An import-time assertion checks that every relation is a proper fraction of its anchor, so the two tables cannot drift apart silently.
• Semantic decoys state a superseded value in equally plausible prose, marked as an earlier revision that was later replaced. Discarding one requires the recency rule in the brief, not a second lookup.
• The join arm plants retry_budget for each service in one file and timeout_ms in a different file far away, then asks which service has the highest timeout_ms among those passing a retry_budget filter. Draws are rejected unless the answer differs from the globally highest timeout_ms, so a model that skips the shallow half and always names the global maximum scores badly instead of well.
• Question order is shuffled away from depth order, so reading the log in sequence earns nothing.

What did 128 runs at up to 730,000 tokens show?

The latent arm held at 100 percent through 34,000 tokens and 99.3 percent at 80,000, then dropped to 94.4 percent at 730,000. The join arm held at 99.2 percent at both lengths it ran. Against NoLiMa's 85 percent retention bar, effective length is at least 700,000 tokens.

The intervals are computed on instances rather than on keys, and that choice costs a lot of apparent precision. Nine keys inside one run share a prompt and a haystack, so they are clustered observations, not independent ones. Version two's 180 key-observations per length were effectively 20, not 180. Counting instances is why a 270 out of 270 result still carries a lower bound of 88.6 percent.

For scale: NoLiMa tested twelve models on a similar latent-association task, and ten of them had fallen below half their short-context baseline by 32,000 tokens. GPT-4.1 posted an effective length of 16,000 against a claimed one million. No Claude model appeared in those tables and nothing in that paper ran past 128,000. A year later, on our version of that task, retention at 730,000 tokens is 94.4 percent.

Why is the smallest context in this study 34,000 tokens?

Because the Claude Code context window is never empty. The CLI ships a system prompt and a full set of tool definitions with every request, and that overhead measured a steady 29,000 tokens across all 128 runs. Our nominal 5,000 token cell is already past the point where most models in the NoLiMa tables had collapsed.

This is the caveat that changes how the whole table reads. We asked for 5,000 tokens of context and measured 33,766. We asked for 50,000 and measured 80,132. The gap is flat, which is what makes it diagnosable as fixed harness overhead rather than anything about the task. It also means our control length is not a short-context baseline in the NoLiMa sense at all. It sits past their 32,000 token column, the column where most of their models had already lost half their accuracy.

The practical version of this: if you run agents through a CLI or an SDK with tools attached, your usable window is smaller than the number on the box, and every request pays that toll before your content arrives. It is worth measuring rather than assuming, and it is a real input to any token budget you write down.

What does a failure at 700,000 tokens actually look like?

Every one of the six latent misses returned the platform base quantity with the service fraction never applied. The model found the shallow half of the pair and stopped. Not one miss across 128 runs grabbed a superseded decoy value, and not one join question ignored its filter.

That is a specific and fairly encouraging failure shape. The model is not confusing itself with stale information, and it is not losing track of the answer's location. It retrieves the first fact, then does not carry it to the second one. Hop one lands and hop two does not. The four misses at 730,000 sit at planted depths of 0.55 to 0.75, inside the same range as the correct answers, so with four data points there is no depth signal to read and we are not going to invent one.

One bookkeeping note, because it changed the scorer. The two misses at 80,000 tokens were originally filed as "other" for the simple reason that no category existed for this failure yet. Both had returned 1,024 for queue_depth, the untouched platform anchor. A miss category that does not exist looks identical to a miss that did not happen, so we added the category and now an assertion at import checks the anchor prose against the anchor numbers, because a desync there would make the new label silently stop matching rather than crash.

Does this refute the 100,000 token dumb zone?

No, and it is not the same claim. Matt Pocock's dumb zone describes a working session degrading as it accumulates its own noise, judged on instruction following and decision quality. This study holds one task fixed and fills the window with inert filler. Different lens, different failure mode.

Pocock's own dictionary states the claim more carefully than the repetitions of it do: early in a session the agent is in a "smart zone", sharp and focused, and as the session grows it drifts into a dumb zone, sloppier, forgetful, more mistakes. Same model, same harness, just more context. It puts the onset around 125K to 150K tokens and adds that this is debated. That hedge is worth preserving. Writing up his own runs against a one million token window he was blunter: "Don't treat 1M context window any differently. It's still 100K of smart, and 900K of dumb." He credits the phrase itself to Dex Horthy.

A working session has three things our prompt does not. It accumulates tool results nobody cleared. It carries failed attempts and abandoned plans nobody swept up. And it feeds the model's own earlier output back in as input. Our haystack is inert: filler written once, never touched, with a fixed question asked cold at the end. That gap is the whole difference between the two experiments, and it is wider than the matching token counts make it look. His own illustration is a session rather than a prompt: an agent stuck on one problem for an hour somewhere around 150K to 200K tokens, which solved it quickly once the context was cleared. Nothing in our design can produce that story, because nothing in our design accumulates.

So the two results sit on different axes rather than in opposition. Where they touch is recall, since the smart zone is described partly as recall being good early, and our data says recall survives to 730,000 tokens in a static prompt. Where they do not touch is most of what the claim is actually about. We never scored instruction adherence and never let a session generate its own noise. Our only failures landed one step past plain retrieval, where two separated facts have to be held together, which suggests composition breaks first.

The experiment that would actually test the dumb zone is the one we have not run: hold a task fixed, grow a real session around it with genuine tool output and genuine dead ends, and score instruction adherence instead of recall. Until someone runs it, read this post as evidence about long prompts and not about long sessions.

What should you change about how you use long context?

Less than the alarm suggests and more than the marketing does, on recall and composition in a fixed task. It says nothing about a long working session. Recall out to 700,000 tokens is good enough that reflexive compaction probably costs more than it saves. What degrades first is composition.

• Measure your own floor before you trust a window size. Harness overhead is real, fixed, and easy to read off a usage field.
• Put facts that have to be combined near each other. The one failure mode we caught is the cross-window join, so co-locating the pieces of an answer removes exactly the thing that broke.
• Treat quadratic attention cost as a separate argument from quality. Advice to keep sessions under 100,000 tokens is largely economics wearing a quality costume, and the two deserve to be priced separately.
• Do not read this as a general verdict. One model, two task shapes, one CLI. Version two proved how easily a well-run experiment can measure the wrong thing and score 100 percent doing it.

The honest summary is that we set out to find context rot, built a probe specifically designed to expose it, spent $83.36 finding out, and got 5.6 points at the far end of a window most tooling will not let you fill. The pre-registration says both outcomes were publishable before we knew which one we had. This is the one we got. The runner, the pre-registration, and every result row are in the repo, and the eight-run cell at 700,000 tokens is the first thing we would spend more on.