What Is an Instruction File Like CLAUDE.md or AGENTS.md?
CLAUDE.md and AGENTS.md are markdown files checked into a repository that an AI coding agent reads before starting any task. AGENTS.md is an open format adopted by over 60,000 GitHub repositories and 28-plus coding tools; Claude Code reads its own CLAUDE.md instead.
An instruction file is a plain markdown file, committed to git, that an AI coding agent reads automatically at the start of a session. It carries the project overview, the commands that build and test the code, and the hard rules the agent must not break. AGENTS.md is the open, tool-agnostic version of this idea, now used in more than 60,000 open-source repositories and natively supported by over two dozen coding tools including Cursor, Codex, and Gemini CLI. Claude Code reads its own CLAUDE.md file instead, which serves the identical purpose. The two formats compete on filename only. The underlying discipline is the same, and it is the subject of this report.
Why Do Agents Ignore Rules That Are Written Down?
An agent has no access to Slack, tribal knowledge, or a decision made verbally last week. If a rule is not written in the repository, it does not exist for the agent, which is why the instinct to keep adding rules to one file feels productive even as it quietly stops working.
An agent has exactly three inputs when it starts a task: the system prompt, whatever files it reads, and whatever a tool returns. It cannot see your Slack thread, your Jira ticket, or the conversation you had with a teammate on Friday. Information that only lives in someone's head does not exist for the agent, full stop. That is precisely why the natural reaction to every mistake, add a rule to the instruction file, feels correct. Writing the rule down is the right instinct. The mistake is where it gets written.
Why Does a Bigger Instruction File Make an Agent Worse?
Past a certain size, an instruction file stops helping. Research on long-context models shows a U-shaped attention curve: models handle information at the start and end of a context far better than the middle, so a hard rule buried at line 300 is frequently ignored even though it is written down.
This is the counterintuitive part. Adding more rules to one growing file does not make an agent more reliable. Past roughly 300 to 600 lines, three things happen at once. First, instruction bloat: a large file eats 10 to 20 percent of the context window before the agent has read a single line of actual code, crowding out the reasoning space a complex task needs. Second, and more interesting, is a documented model behavior called lost in the middle. Liu et al. (2023) tested how well language models retrieve information depending on where it sits in a long context, and found a consistent U-shaped curve: models handle information placed at the very start or very end of the context far better than information buried in the middle, and the effect held even for models built specifically for long-context tasks. A hard security rule sitting at line 300 of a 600-line file is exactly the kind of instruction that finding proves gets ignored.
| Position of the answer in a 20-document context | GPT-3.5 accuracy |
|---|---|
| Document 1 (start of context) | ~75% |
| Document 10 (middle of context) | ~55% |
| Document 20 (end of context) | ~72% |
Third, priority conflicts creep in. A hard constraint like "never run destructive commands" and a soft preference like "prefer short functions" end up formatted identically, so the agent has no signal for which one is actually a red line. All three problems compound the same way instruction bloat compounds anywhere: nobody deletes old rules, because deletion always feels riskier than addition, so the signal-to-noise ratio drops every single week.
What Is the Router Pattern for Instruction Files?
Keep the entry file to 50 to 200 lines: project overview, first-run commands, a short list of hard constraints, and one-line pointers to topic docs. Detailed rules live in separate files the agent opens only when the task actually needs them.
The fix is called progressive disclosure, and it treats the entry file as a router instead of an encyclopedia. AGENTS.md or CLAUDE.md stays short: what the project is, how to run it, how to verify it, and no more than about fifteen hard, non-negotiable constraints. Everything else moves into topic docs of 50 to 150 lines each, sitting in the module or directory they actually apply to: a database rules file next to the database code, an API patterns file next to the API layer. The router links to each one with a single line explaining when it applies, so the agent pulls it in only when the task actually touches that area. This is the exact structure we install as part of every harness engineering build, because an agent's context window is a scarce resource, not a place to store an entire company's tribal knowledge.
Matt Pocock, who has published some of the most widely used practical guidance on this pattern, frames the entry file the same way: small, focused, and pointing elsewhere, giving the agent just enough to start and letting deeper context load on demand through nested files or skills. He also names a useful budget: frontier thinking models follow roughly 150 to 200 discrete instructions with real consistency, and smaller or non-reasoning models handle fewer. Treat that number as a hard ceiling on your entry file, not a target to fill.
This is not just a style preference. A 2026 study out of King's College London and the University of Passau measured it directly: across 10 repositories and 124 real pull requests, running the same coding agents with and without an AGENTS.md file present, the file's presence cut median agent runtime by 28.64 percent and output token consumption by 16.58 percent, while task completion stayed comparable. A well-scoped instruction file does not just prevent mistakes. It measurably speeds the agent up.
| Metric (with AGENTS.md vs. without, same tasks) | Measured effect |
|---|---|
| Median agent runtime | 28.64% lower |
| Output token consumption | 16.58% lower |
| Task completion behavior | Comparable, no regression |
Why Should Instructions Follow One Concept Per File?
A topic doc that mixes deploy steps, style preferences, and one specific bug fix from last quarter forces the agent to load all three every time it needs any one of them. One concept per file means a task only pulls in the context it actually needs.
The one-concept-per-file rule is what makes progressive disclosure actually work instead of just relocating the same bloat into more files. If your database rules file also contains deploy instructions and a paragraph about a WebSocket bug from March, an agent working on a schema migration loads all of it, including the two-thirds that is irrelevant to the task in front of it. Split by concept: one file for database constraints, one for API patterns, one for deploy steps. A historical bug note is not an instruction at all. It should become a test case that fails if the bug comes back, not a permanent paragraph a future session has to keep reading.
How Do You Know If Your Instruction File Is Actually Working?
Run the fresh-session test: open a brand-new agent session with nothing but the repository, and check whether it can answer five questions without guessing. Any blank is a gap in the map, not a gap in the agent.
There is a simple test that replaces guesswork with evidence. Open a completely fresh agent session, point it at the repository, and nothing else, then check whether it can answer five questions on its own: what is this system, how is it organized, how do I run it, how do I verify it, and where are we right now. Every blank answer marks a spot where the agent has to guess, and a wrong guess becomes a bug that a human then has to trace back to a documentation gap. This test costs about the same as writing one prompt and it is the single fastest way to find out whether your instruction file is a router or just a pile of good intentions.
Why Should Instructions Live in the Repo Instead of a Wiki?
A wiki page, a Notion doc, or a Slack thread is invisible to an agent unless something actively puts it in front of the model. The repository is the one place an agent reads by default at the start of every session, which makes it the only reliable source of truth for a working system.
Confluence pages go stale because updating them is optional and disconnected from the code. A rule sitting in the repository, next to the code it governs, gets updated in the same commit as the code change that made it necessary, or it visibly rots where anyone can see it happened. Keeping instructions in the repo is not a stylistic preference. It is the only version of "documented" an agent can actually act on, since a wiki link the agent never opens carries the same practical weight as a rule that was never written down at all. This same principle drives how we structure state and progress files in every custom AI agent we build: if it isn't in a file the agent will actually read, we don't count on it.
The payoff compounds the same way in a long-running autonomous coding loop: an agent working overnight without a human in the room has nothing but the repository to orient itself session after session, so a router-style instruction file is what keeps an unattended run from drifting.
