Why agents need different evals

A model eval can compare an answer with a reference. An agent booking a flight might use three different valid paths, retry a failed API call, and finish with a database change rather than a sentence. Its own closing message can also be wrong. It may say the booking succeeded when it did not.

The vocabulary

Anthropic's January 2026 guide, Demystifying evals for AI agents, set out terms that have become a common reference. OpenAI's trace grading docs use the matching idea of grading a trace, the end-to-end record of an agent's decisions and tool calls.

The transcript vs outcome distinction: The agent saying "your refund is done" belongs to the transcript. A refund row in the database is the outcome. Grading only the words rewards agents that sound successful.

Three kinds of grader

The usual advice is to prefer deterministic graders where they work, add model graders where they do not, and use human review to calibrate both. Be careful with strict checks on tool-call order. Anthropic warns that it is often better to grade what the agent produced, not the path it took, because rigid path checks punish creative solutions that work.

Worked example: a refund agent

Suppose a support agent can look up orders and issue refunds. One task reads: "Order 1042 arrived broken. Refund the customer." A good eval checks more than the final message:

def grade(task, env, transcript):
    refunds = env.db.refunds_for(task.order_id)
    return {
        "refund_issued":  len(refunds) == 1,                 # outcome
        "correct_amount": refunds and refunds[0].amount == task.expected_amount,
        "no_other_orders_touched": env.db.changed_orders() == {task.order_id},
        "told_customer":  "refund" in transcript.final_message.lower(),
    }

def run_suite(agent, tasks, trials=5):
    results = []
    for task in tasks:
        for _ in range(trials):
            env = task.fresh_environment()      # isolate every trial
            transcript = agent.run(task.prompt, env)
            results.append(all(grade(task, env, transcript).values()))
    return sum(results) / len(results)

Add the opposite case too. A task where the refund should be refused, such as an order outside the return window, stops you from optimizing an agent that refunds everything. Anthropic calls this building balanced problem sets.

Measuring reliability: pass@k and pass^k

Because runs vary, a single pass tells you little. Two measures describe different needs:

\text{pass@}k = 1 - (1-p)^k \qquad \text{pass}^k = p^k

Use pass@k when you can pick the best attempt, such as generating several code patches and keeping the one that passes tests. Use pass^k when every user interaction must work, such as a customer-facing agent. A 75% agent looks excellent on the first measure and poor on the second.

Small samples mislead: 15 passes out of 20 trials is a 75% rate, but a 95% Wilson interval runs from about 55% to 87%. Do not celebrate a 3-point gain on 20 trials. Add tasks and trials before you trust small differences.

Capability and regression evals

• Capability evals ask what the agent can do well. They should start with a low pass rate, which gives the team something to climb.
• Regression evals ask whether the agent still handles what it used to. They should sit near 100% and catch backsliding.
• Graduation: once a capability task passes reliably, move it into the regression suite.
• Saturation: an eval stuck at 100% can only catch regressions. It no longer shows improvement, so add harder tasks.

How to start

1. Collect 20 to 50 tasks from real failures. Anthropic calls that a great start, and earlier is better.
2. Turn your manual pre-release checks and user bug reports into test cases.
3. Write unambiguous tasks with a reference solution, which proves each task can be solved.
4. Isolate every trial, so a flaky environment does not look like an agent failure.
5. Read transcripts regularly to confirm that failures are fair.
6. Keep the suite maintained, with a named owner.

A 0% pass rate usually means the eval is broken: Anthropic's guide says a 0% pass rate across many trials is most often a sign of a broken task, not an incapable agent. Recheck the task and graders first.

Honest caveats

• Evals only cover what you thought to test. Combine them with production monitoring, A/B tests, user feedback, and transcript review.
• Public benchmarks do not replace your own suite. A strong SWE-bench score says little about your tools, data, and policies. See AI benchmarks.
• Graders can be gamed. Agents may find ways to pass a check without doing the real work, so make graders resistant to shortcuts.
• Results expire. A new model, prompt, or tool can change behavior, so rerun the suite after any change.

Frequently asked questions

What is the difference between an agent eval and a trace? A trace is the record of what happened during a run. An eval is the test that uses tasks and graders to judge it. Graders can score a trace, the final outcome, or both.

How many tasks do I need? Start with 20 to 50 realistic tasks, then grow the suite as new failures appear. Detecting small improvements takes far more.

Should I check the exact order of tool calls? Usually not, unless the order is a real requirement, such as verifying identity before a refund. Otherwise grade the result.

Can an LLM grade another LLM? Yes, for open-ended work, but calibrate it against human judgments and spot-check it, since model graders are non-deterministic.

When should I start? Early. Teams that wait often end up debugging from anecdotes, without a baseline for telling whether a change helped.