AI Agent Evaluation: Test the System, Not the Model

Why Do Benchmark Scores Not Transfer to Production Agents?

Benchmark scores don't transfer because they measure a model under curated conditions, while a production agent is a full system of model, prompts, tools, and data. The same model can succeed in one system and fail in another, so a leaderboard rank is useful only for shortlisting candidates and cannot predict whether your specific agent works.

A public benchmark holds everything constant except the model, with curated tasks, clean inputs and a fixed scoring script. A production agent has none of that: it runs on your messy documents, calls your tools with your descriptions of them, follows your instructions file, and serves a goal the benchmark authors never imagined. Two systems built on the same model can sit at opposite ends of the reliability spectrum, and we have seen the weaker model outperform the stronger one inside a better-built system. The public record shows the same effect at a larger scale. When SWE-bench launched, the best frontier model resolved just 1.96 percent of its real GitHub issues, and within a year SWE-agent lifted that to 12.5 percent using the same generation of models, purely by redesigning the interface the model worked through.

In our process, benchmarks are used for one thing, which is shortlisting candidate models. Everything after that is system evaluation, run on your tasks and in your environment against your definition of success. The interface lesson applies to failures too. Most agent failures trace to what the model saw rather than what it could do, which is the subject of our report on context engineering.

How Do You Evaluate an AI Agent?

Evaluate an AI agent in three layers: static checks that outputs parse and validate, task-level tests where every task has a command-checkable definition of done, and end-to-end runs of the full pipeline as a real user would trigger it. The system passes only when all three layers pass.

We evaluate agents the way disciplined teams test software, in three layers that catch different classes of failure. A system passes when all three layers pass, however good the demo looked.

1. Static checks. Before judging behavior, verify structure: outputs parse, schemas validate, required fields exist, and code lints and type-checks. These checks are cheap and instant, and they catch the failures that should never reach a human reviewer.
2. Task-level tests with verification commands. Every task the agent performs gets a command-checkable definition of done, such as "the test suite passes," "the file exists at this path with these fields," or "the reply contains no banned phrases per the linter." "The summary looks good" does not qualify. If success cannot be expressed as a command someone can run, the task is not specified well enough to evaluate.
3. End-to-end runs. The full pipeline is executed as a real user would trigger it, from inbound request to delivered outcome. Task-level tests cannot see boundary defects, because every component can pass alone while the handoffs between them fail, so end-to-end runs are the only proof of system-level correctness.

Most teams stop after a casual version of layer two, but almost every painful production surprise we have investigated was in layer three.

Where Should the Eval Set Come From?

Evaluation tasks should come from the real workflow the agent will run: actual emails, documents, and tickets, anonymized where needed. Twenty genuine cases, including the messy ones, beat two hundred invented ones, because synthetic tests inherit the blind spots of whoever wrote the prompts. Add every production failure back into the set.

We pull test tasks from the actual workflow the agent will run instead of making them up. Synthetic test cases written by the same people who wrote the prompts inherit the same blind spots, so they pass tasks the real world fails. A small eval set of twenty genuine cases, including the ugly ones with missing fields and contradictory instructions, beats two hundred invented ones. When we evaluate an AI receptionist build, for example, the eval set is made of real call transcripts and after-hours messages rather than scenarios we wrote for it to pass. Every production failure that slips through later gets added to the set, so the suite keeps getting better instead of going stale.

Why Must the Checker Be Separate From the Worker?

The checker must be separate because agents systematically over-grade their own output, claiming success whether the work is excellent or subtly broken, so self-report has to be treated as noise. Use a deterministic script or a separate adversarial agent as the grader, and let only verification output mark a task as passing.

The most consistent finding from our lab work is that agents over-grade their own output. Ask an agent whether it finished the job and it says yes with the same confidence whether the work is excellent, half-done, or subtly broken. It happens systematically enough that we treat self-report as noise.

We treat an agent's claim that the work is done as a hypothesis, and only a verifier the agent does not control can promote it to a result.

Public reliability data points the same way. On tau-bench, which tests tool-using agents against simulated customers under policy rules, GPT-4o completes fewer than half the tasks on a single attempt, and Sierra's analysis reports its retail-domain score dropping to roughly 25 percent when the same task must pass eight runs in a row. Passing once does not show that an agent will keep passing.

In practice, the entity that grades is never the entity that did the work. The checker is sometimes a deterministic script and sometimes a second agent prompted adversarially, which is instructed to refute the claim that the work is done rather than confirm it. In our task tracking, the worker can mark a task ready for review but can never mark it passing, because only verification output flips that state.

What Counts as Evidence That an Agent Works?

Evidence means auditable artifacts: the exact command that ran, its output, the commit hash, and timestamps. A claim like it worked when we tried it cannot be audited weeks later. Every passing evaluation should leave a record a third party could inspect and reproduce without trusting anyone's memory.

Every passing evaluation produces artifacts: the command that ran, its output, the commit hash, and timestamps. Nobody can audit "it worked when we tried it" three weeks later. When a client asks why we trust an agent with a workflow, we show them a folder of that evidence. The standard is built into every system we ship through our custom AI agent engagements, because the client inherits the evaluation harness along with the agent.

What Is the Difference Between Outputs and Outcomes?

Outputs are what the agent produces; outcomes are what that work achieves. An agent can pass every test and still write emails nobody answers or reports nobody reads. Production agents need outcome instrumentation such as response rates, resolution times, human edit rates, and downstream conversions, because the final test is whether the real number moves.

No test command reaches the last layer of evaluation. An agent can produce flawless outputs that achieve nothing, such as emails that pass every style check and never get a reply, reports nobody reads, or tickets routed correctly into a queue nobody works. The three layers verify output quality, while outcome quality is whether the work moved the number the system exists to move.

So production agents get outcome instrumentation: response rates, resolution times, human edit rates on agent drafts, and downstream conversions. The edit rate has been an especially trustworthy signal for us. When humans stop correcting an agent's drafts, that is a stronger pass than any benchmark score, and when edits climb, something has drifted no matter what the tests say. The same instrumentation belongs in any workflow automation pipeline, agent-driven or not, because a pipeline that runs perfectly toward the wrong number has still failed.

How Should a Team Start Evaluating AI Agents Today?

Start by writing the verification command for one task you actually need done, before you ask which model is best. Build the smallest agent that can attempt the task, run static checks, task tests, and an end-to-end pass, and let the evidence decide. Teams that struggle are the ones that never defined what passing means.

Begin by writing the verification command for one task you actually need done, and leave the question of which model is best for later. Build the smallest agent that can attempt it, run the three layers, and let the evidence decide. The teams that struggle are the ones who never wrote down what passing means, whichever model they picked. If you are not sure which task to start with, an AI readiness assessment is built to answer that question.

Evaluation also continues after launch. The eval suite is the harness the agent lives inside for as long as it runs, the same way a test suite outlives the sprint that wrote it. We treat it as a deliverable with the same standing as the agent itself, and our report on harness engineering covers that approach in full.