AI Agent Evaluation: Test the System, Not the Model

Why Do Benchmark Scores Not Transfer to Production Agents?

Benchmark scores measure a model under curated conditions, while a production agent is a full system of model, prompts, tools, and data. The same model can succeed in one system and fail in another, so a leaderboard rank only shortlists candidates. It never predicts whether your specific agent works.

A public benchmark holds everything constant except the model: curated tasks, clean inputs, a fixed scoring script. Your production agent holds nothing constant. It runs on your messy documents, calls your tools with your descriptions of them, follows your instructions file, and serves a goal the benchmark authors never imagined. Two systems built on the same model can sit at opposite ends of the reliability spectrum, and we have seen the weaker model outperform the stronger one inside a better-built system. The public record shows the same effect at scale: when SWE-bench launched, the best frontier model resolved just 1.96 percent of its real GitHub issues, and within a year SWE-agent lifted that to 12.5 percent using the same generation of models, purely by redesigning the interface the model worked through.

So benchmarks earn exactly one job in our process: shortlisting candidate models. Everything after that is system evaluation, run on your tasks, in your environment, against your definition of success. The interface lesson cuts the other way too: most agent failures trace to what the model saw, not what the model could do, which is the subject of our report on context engineering.

How Do You Evaluate an AI Agent?

Evaluate an AI agent in three layers: static checks that outputs parse and validate, task-level tests where every task has a command-checkable definition of done, and end-to-end runs of the full pipeline as a real user would trigger it. The system passes only when all three layers pass.

We evaluate agents the way disciplined teams test software, in three layers that catch different failure classes. A system passes when all three pass, not when a demo looks good.

1. Static checks. Before judging behavior, verify structure: outputs parse, schemas validate, required fields exist, code lints and type-checks. These checks are cheap, instant, and catch the failures that should never reach a human reviewer.
2. Task-level tests with verification commands. Every task the agent performs gets a command-checkable definition of done. Not "the summary looks good" but "the test suite passes," "the file exists at this path with these fields," "the reply contains no banned phrases per the linter." If success cannot be expressed as a command someone can run, the task is not specified well enough to evaluate.
3. End-to-end runs. The full pipeline, executed as a real user would trigger it, from inbound request to delivered outcome. Task-level tests are blind to boundary defects: every component can pass alone while the handoffs between them fail. End-to-end runs are the only proof of system-level correctness.

Most teams stop after a casual version of layer two. Almost every painful production surprise we have investigated lived in layer three.

Where Should the Eval Set Come From?

Evaluation tasks should come from the real workflow the agent will run: actual emails, documents, and tickets, anonymized where needed. Twenty genuine cases, including the messy ones, beat two hundred invented ones, because synthetic tests inherit the blind spots of whoever wrote the prompts. Add every production failure back into the set.

We pull test tasks from the actual workflow the agent will run, never from imagination. Synthetic test cases written by the same people who wrote the prompts inherit the same blind spots, so they pass tasks the real world fails. A small eval set of twenty genuine cases, including the ugly ones with missing fields and contradictory instructions, beats two hundred invented ones. When we evaluate an AI receptionist build, for example, the eval set is real call transcripts and after-hours messages, not scenarios we wrote for it to ace. Every production failure that slips through later gets added to the set, which is how the eval suite compounds instead of staling.

Why Must the Checker Be Separate From the Worker?

Agents systematically over-grade their own output, claiming success whether the work is excellent or quietly broken, so self-report has to be treated as noise. The entity that grades must never be the entity that worked. Use a deterministic script or a separate adversarial agent, and only verification output can mark a task as passing.

The most consistent finding from our lab work: agents over-grade their own output. Ask an agent whether it finished the job and it says yes with the same confidence whether the work is excellent, half-done, or quietly broken. This is not occasional. It is systematic enough that we treat self-report as noise.

An agent's claim that the work is done is a hypothesis, not a result. Only a verifier the agent does not control can promote it.

Public reliability data points the same direction. On tau-bench, which tests tool-using agents against simulated customers under policy rules, GPT-4o completes fewer than half the tasks on a single attempt, and Sierra's analysis reports its retail-domain score dropping to roughly 25 percent when the same task must pass eight runs in a row. An agent that passes once is not yet an agent that passes.

Operationally that means the entity that grades is never the entity that worked. Sometimes the checker is a deterministic script. Sometimes it is a second agent prompted adversarially, instructed to refute the claim that the work is done rather than confirm it. In our task tracking, the worker can mark a task ready for review but can never mark it passing. Only verification output flips that state.

What Counts as Evidence That an Agent Works?

Evidence means auditable artifacts: the exact command that ran, its output, the commit hash, and timestamps. A claim like it worked when we tried it cannot be audited weeks later. Every passing evaluation should leave a record a third party could inspect and reproduce without trusting anyone's memory.

Every passing evaluation produces artifacts: the command that ran, its output, the commit hash, timestamps. "It worked when we tried it" is not a record anyone can audit three weeks later. When a client asks why we trust an agent with a workflow, the answer is a folder of evidence, not a feeling. This standard is built into every system we ship through our custom AI agent engagements, because the client inherits the evaluation harness along with the agent.

What Is the Difference Between Outputs and Outcomes?

Outputs are what the agent produces; outcomes are what that work achieves. An agent can pass every test and still write emails nobody answers or reports nobody reads. Production agents need outcome instrumentation: response rates, resolution times, human edit rates, and downstream conversions, because moving the real number is the final test.

There is a final layer that no test command reaches. An agent can produce flawless outputs that achieve nothing: emails that pass every style check and never get a reply, reports nobody reads, tickets routed correctly into a queue nobody works. Output quality is what the three layers verify. Outcome quality is whether the work moved the number the system exists to move.

So production agents get outcome instrumentation: response rates, resolution times, human edit rates on agent drafts, downstream conversions. The edit rate has been especially honest with us. When humans stop correcting an agent's drafts, that is a stronger pass than any benchmark score, and when edits climb, something drifted no matter what the tests say. The same instrumentation belongs in any workflow automation pipeline, agent-driven or not, because a pipeline that runs perfectly toward the wrong number is still a failure.

How Should a Team Start Evaluating AI Agents Today?

Start by writing the verification command for one task you actually need done, not by asking which model is best. Build the smallest agent that can attempt the task, run static checks, task tests, and an end-to-end pass, then let the evidence decide. Teams fail by never defining what passing means.

Do not begin by asking which model is best. Begin by writing the verification command for one task you actually need done. Build the smallest agent that can attempt it, run the three layers, and let the evidence decide. The teams that struggle are not the ones with weak models. They are the ones who never wrote down what passing means. If you are not sure which task to start with, that selection problem is exactly what an AI readiness assessment is for.

Evaluation is also not a phase you finish. It is the harness the agent lives inside for as long as it runs, the same way a test suite outlives the sprint that wrote it. We treat the eval suite as a deliverable with the same standing as the agent itself, and the full doctrine around that is in our report on harness engineering.