What Exactly Can an AI Agent See When It Works?
An AI agent sees only three inputs: the system and task prompt, the contents of files it can read, and the output of tools it runs. Everything else, including meetings, chat threads and tribal knowledge, is invisible to it unless someone writes it down.
This is easy to say and hard to internalize, because people on a team constantly fill gaps with context nobody wrote down. A senior engineer knows the payments service can't be touched on Fridays because of a settlement job, and that fact lives in her head, which an agent working the same repo has no access to. If the constraint isn't in a file the agent reads, the agent will touch the payments service on a Friday, and that will be the predictable result of a map with a hole in it rather than an isolated mistake. We treat this as a design constraint on every custom AI agent we build: if a rule matters, it goes in a file the agent actually reads before it acts.
What Is the Fresh-Session Test and How Do You Run It?
The fresh-session test checks whether a brand-new session, given only the repo, can answer five questions: what is this system, how is it organized, how do I run it, how do I verify it, and where are we right now. Any question it cannot answer marks a real gap.
The test is deliberately strict. Don't answer the agent's questions yourself or paste in extra context from your own memory; watch what a cold session can work out from the files on disk alone. Five specific questions decide the grade: what is this system, how is it organized, how do I run it, how do I verify it, and where does the work currently stand. Every blank the agent hits is a guess it will make on your behalf, and wrong guesses turn into bugs as well as wasted time. The test also grades human onboarding, since a repo that fails it is usually the same repo where a new hire spends their first two weeks asking questions nobody wrote down.
Why Does an Instructions File Stop Working Once It Gets Long?
Long instruction files get worse past a few hundred lines, because models use text at the start and end of a long context far more reliably than text buried in the middle. A hard rule at line 300 of such a file is easy to miss entirely.
The natural response to every agent mistake is "add a rule to the instructions file." That works the first few times, but once the file hits four or five hundred lines it becomes worse than useless, because the agent gets measurably worse at following it. Liu and colleagues documented the mechanism directly. Testing how language models use long inputs, they found performance is highest when the relevant information sits at the beginning or end of the context and drops off sharply when it is buried in the middle, a pattern that holds even in models built specifically for long context. A security rule at line 300 of a bloated instructions file is therefore a serious risk, since that is statistically likely to be the spot the model pays least attention to. We cover the layer this problem sits inside, and how it interacts with the rest of an agent's context, in our report on context engineering.
| Position of relevant information in the context | Retrieval accuracy pattern observed |
|---|---|
| Beginning of a long input | High, models retrieve it reliably |
| Middle of a long input | Significantly degraded, even in models designed for long context |
| End of a long input | High, models retrieve it reliably |
Make the entry file a router that points elsewhere. Keep it short (a project overview, hard constraints and pointers to topic docs) and let the agent pull detailed docs on demand rather than scrolling past all of them on every task. A well-organized website works the same way: the homepage points to the other pages instead of holding all their content.
Does More Context Actually Mean More Reliable Work?
No. More context and more tool calls raise cost fast without a matching reliability gain unless the extra information is relevant. Anthropic's own multi-agent research system uses roughly fifteen times the tokens of a single chat turn to earn its performance gain, and that ratio is a deliberate design choice.
It is tempting to assume that giving an agent more information is always safer than giving it less. Anthropic's writeup of their multi-agent research system is a useful check on that, because they measured the tradeoff directly instead of assuming it. A single agent uses roughly four times the tokens of a normal chat exchange, and a multi-agent system, where separate agents each carry their own context window, uses roughly fifteen times as many. In their case the extra cost bought a real performance gain, with a multi-agent architecture outperforming a single agent by over 90% on their internal evaluation. But they were explicit that token usage alone explained about 80% of the variance in outcome, meaning most of what mattered was whether the agent had access to the right information, structured well, rather than just more of it stacked on top.
| System architecture | Token usage relative to a single chat turn |
|---|---|
| Single chat exchange | 1x baseline |
| Single agent using tools | ~4x |
| Multi-agent system, separate context windows | ~15x |
Information that is not in the repo does not exist for the agent, so what matters is whether you drew a good enough map to the context you have.
What Does a Repo That Passes the Fresh-Session Test Actually Look Like?
A repo that passes has a short entry file with hard constraints and pointers, topic docs next to the code they describe, a progress file updated every session, and a decisions file recording what was chosen and why. Nothing important lives only in a person's memory.
In practice this is four ordinary kinds of file working together. A short entry file at the root carries the overview, the run commands and the non-negotiable constraints. Topic docs sit next to the code they govern instead of in a wiki nobody updates. A progress file, updated at the end of every session, records what is done, what is in flight and what is blocked. A decisions file records what was chosen, why, and what was rejected, because the code shows what was built but not the reasons for it. With that structure in place, a new session (human or agent) can get its bearings in a few minutes instead of guessing its way through the first hour. We put the same backbone under every automation build before letting an agent touch it unsupervised, and it is worth doing on projects a human will pick back up too, because the fresh-session test works the same way whether the next reader is a person or an agent.
What Happens When a Team Skips This and Relies on Memory Instead?
When a team relies on memory, rules that live only in chat threads, a stale wiki or one engineer's head are by definition invisible to the agent. It cannot tell a rule that still matters from one that stopped applying last quarter, because neither appears anywhere it can read.
Across the client builds we have run, this failure is consistent enough to recognize on sight. Protocol decisions live in old Slack threads that search can't surface, a wiki page describes an API that changed two quarters ago, and one senior engineer is the only person who remembers why a particular service can't be redeployed on a Friday. An agent dropped into that environment has no way to distinguish a rule that still matters from one that has lapsed, because neither is written anywhere it reads. Fixing this does not take a heroic documentation sprint. Close the specific gaps the fresh-session test exposes, one blank at a time, and then keep the habit by updating the relevant file in the same commit as the code change it describes, instead of in a separate pass that is easy to skip under deadline pressure.
