What Exactly Can an AI Agent See When It Works?
An agent's entire world is three inputs: the system and task prompt, the contents of files it can read, and the output of tools it runs. Everything else, meetings, chat threads, tribal knowledge, is invisible to it by default.
This is easy to say and hard to internalize, because humans on a team constantly fill gaps with context nobody wrote down. A senior engineer knows the payments service can't be touched on Fridays because of a settlement job; that fact lives in her head. An agent working the same repo has no access to her head. If the constraint isn't in a file the agent reads, the agent will happily touch the payments service on a Friday, and it won't be an isolated mistake, it'll be the predictable result of a map with a hole in it. We treat this as a design constraint on every custom AI agent we build: if a rule matters, it goes in a file the agent actually reads before it acts.
What Is the Fresh-Session Test and How Do You Run It?
Open a brand-new session, hand it only the repo, and see if it can answer five questions: what is this system, how is it organized, how do I run it, how do I verify it, and where are we right now. Any blank is a real gap.
The test is deliberately brutal. Don't answer the agent's questions yourself. Don't paste in extra context from your own memory. Just watch what a cold session, with nothing but the files on disk, can figure out on its own. Five specific questions decide the grade: what is this system, how is it organized, how do I run it, how do I verify it, and where does the work currently stand. Every blank the agent hits is a guess it will make on your behalf, and wrong guesses become bugs, not just wasted time. The uncomfortable part is that this test also grades human onboarding. A repo that fails the fresh-session test is usually the same repo where a new hire spends their first two weeks asking questions nobody wrote down.
Why Does an Instructions File Stop Working Once It Gets Long?
Long instruction files get worse, not better, past a few hundred lines. Models use text at the start and end of a long context far more reliably than text buried in the middle, so a hard rule at line 300 is easy to miss entirely.
The natural response to every agent mistake is "add a rule to the instructions file." It works the first few times. Then the file hits four or five hundred lines and something worse than uselessness happens: the agent gets measurably worse at following it. Liu and colleagues documented the mechanism behind this directly. Testing how language models use long inputs, they found performance is highest when the relevant information sits at the beginning or end of the context, and drops off sharply when it's buried in the middle, a pattern that holds even in models built specifically for long context. A security rule sitting at line 300 of a bloated instructions file isn't a minor risk. It's statistically likely to be the exact spot the model pays least attention to. We cover the layer this problem sits inside, and how it interacts with the rest of an agent's context, in our report on context engineering.
| Position of relevant information in the context | Retrieval accuracy pattern observed |
|---|---|
| Beginning of a long input | High, models retrieve it reliably |
| Middle of a long input | Significantly degraded, even in models designed for long context |
| End of a long input | High, models retrieve it reliably |
The fix is a router, not an encyclopedia. Keep the entry file short, a project overview, hard constraints, and pointers to topic docs, and let the agent pull detailed docs on demand rather than scrolling past all of them on every task. It's the same reason a well-organized site doesn't put every page's content on the homepage; a good map points, it doesn't contain.
Does More Context Actually Mean More Reliable Work?
No. More context and more tool calls raise cost fast without a matching reliability gain unless the extra information is genuinely relevant. Anthropic's own multi-agent research system uses roughly fifteen times the tokens of a single chat turn to earn its performance gain, and that ratio is deliberate, not incidental.
It's tempting to assume that giving an agent more information is always safer than giving it less. Anthropic's writeup of their multi-agent research system is a useful reality check, because they measured the tradeoff directly instead of assuming it. A single agent uses roughly four times the tokens of a normal chat exchange; a multi-agent system, with separate agents each carrying their own context window, uses roughly fifteen times as many. That additional cost bought a real performance gain in their case, a multi-agent architecture outperformed a single agent by over 90% on their internal evaluation, but they were explicit that token usage alone explained about 80% of the variance in outcome, meaning most of what mattered was whether the agent had access to the right information, structured well, not just more of it stacked on top.
| System architecture | Token usage relative to a single chat turn |
|---|---|
| Single chat exchange | 1x baseline |
| Single agent using tools | ~4x |
| Multi-agent system, separate context windows | ~15x |
Information not in the repo does not exist for the agent. The question is never whether you have context. It's whether you drew a good enough map to it.
What Does a Repo That Passes the Fresh-Session Test Actually Look Like?
A short entry file with hard constraints and pointers, topic docs living next to the code they describe, a progress file updated every session, and a decisions file recording what was chosen and why. Nothing load-bearing lives only in a person's memory.
In practice this is four kinds of file working together, none of them exotic. A short entry file at the root carries the overview, the run commands, and the non-negotiable constraints. Topic docs sit next to the code they govern instead of in a wiki nobody updates. A progress file, updated at the end of every session, records what's done, what's in flight, and what's blocked. A decisions file records what was chosen, why, and what was rejected, because the code alone shows the what and loses the why. Get that structure in place and a new session, human or agent, can rebuild its bearings in a few minutes instead of guessing its way through the first hour. This is the same backbone we put under every automation build before letting an agent touch it unsupervised, and it's worth doing even on projects a human will pick back up, because the fresh-session test doesn't actually care whether the next reader has a pulse.
What Happens When a Team Skips This and Relies on Memory Instead?
Rules that only live in chat threads, a stale wiki, or one engineer's memory are invisible to an agent by definition. It has no way to tell a rule that still matters from one that quietly died last quarter, because both look identical from where it sits: absent.
The failure mode is consistent enough to recognize on sight across the client builds we've run. Protocol decisions live in old Slack threads search can't surface. A wiki page describes an API that changed two quarters ago. One senior engineer is the only person who remembers why a particular service can't be redeployed on a Friday. An agent dropped into that environment has no way to distinguish a rule that still matters from one that quietly died, because both look identical from where it sits: absent. The fix isn't a heroic documentation sprint. It's closing the specific gaps the fresh-session test exposes, one blank at a time, and then keeping the habit alive by updating the relevant file in the same commit as the code change it describes, not in a separate pass that's easy to skip under deadline pressure.
