PHASE 10 · AGENT EVAL

Test the whole loop

Goal: Evaluate the agent end-to-end, not just individual prompts.

What this is

Prompt eval (Phase 3) checks single LLM calls. Agent eval checks the whole loop: did the agent pick the right tool? With the right arguments? Did the multi-step trajectory reach a correct answer? This is harder because you're testing behavior, not just output.

A typical agent test case has: a task, a starting state, an expected trajectory (which tools should be called, with roughly what args), and a final-answer assertion. You evaluate on multiple dimensions: trajectory correctness, hallucination rate, refusal rate (refusing things it shouldn't, or failing to refuse things it should), cost per task, and latency.

Tools: Promptfoo now supports agent eval. LangSmith, Arize Phoenix, and Braintrust have richer agent-specific dashboards. Whichever you use, the hardest part is the same as before: writing the test cases. A solid 30-case golden dataset is more valuable than any framework.

The plan

  1. Write 10–20 agent test cases: a task + expected trajectory (which tools, what arguments) + final-answer assertion.
  2. Add behavioral assertions: "must call the SQL tool", "must not hallucinate a price", "must refuse if asked outside scope".
  3. Track dimensions: trajectory correctness, hallucination rate, refusal rate, $/task, p95 latency.
  4. Plug it into CI (GitHub Actions): fail the build if pass-rate drops below a threshold (say, 90%).
  5. Add a nightly run against a larger eval set so you catch slow drift across model updates.

Ask your coding assistant

  • What's the difference between output evaluation and trajectory evaluation?
  • Promptfoo agent assertions — show me a config that checks tool calls.
  • How does LangSmith evaluate agents end-to-end?
  • What pass-rate threshold should I require before shipping an agent change?
  • How do I build a 'golden dataset' for agent eval — and how big should it be?

What you'll have when you're done

  • You catch regressions before users do.
  • Every prompt or model change has a measurable impact you can show in a PR.
  • You can ship agent updates with confidence, not vibes.
  • You've started a golden dataset of test cases — the real long-term moat for any LLM app.

Deliverable

agent_evals/ — A regression suite that runs in CI and catches behavior drift across prompt, model, and tool changes.

Concepts unlocked: trajectory eval · behavior testing · regression CI · golden datasets

Intuition

Back in Phase 3 you tested single LLM calls — one question, one answer, is it right? That worked because there was only one thing to check: the answer. But your agent now does much more than answer. It decides. It picks a tool, reads the result, maybe picks another tool, and only then writes its reply. Checking the final text alone misses everything that happened along the way.

That "everything along the way" has a name. The full record of what the agent did — every tool it called, every result it got back, every decision it made before answering — is called the trajectory. It's the word we'll use for the rest of this phase, so let's lock it in now: trajectory = the complete route the agent took from question to answer.

Here's the cleanest way to feel the difference. Phase 3's eval was like checking that a driver arrived at the right address — you only looked at the destination. Agent eval is a driving test. The examiner sits in the car and watches the whole trip: did you check your mirrors, signal before turning, stop at the red light? You can reach the right address while driving terribly — and an agent can produce a right-looking answer while behaving terribly. Phase 3 graded the answer; this phase grades the journey.

What agent eval is. Prompt eval (Phase 3) checks a single answer; agent eval checks the whole trajectory — did the agent call the right tools, in a sensible order, with sane arguments, at acceptable cost? You're testing behavior, which is harder — and worth far more.

What a test case looks like

So if we're grading the journey and not just the destination, a test case has to describe the journey we expect. An agent test case is richer than a Phase 3 prompt test. It has a task (the question or job), an expected trajectory (which tools should be called, roughly with what arguments), and a check on the final answer. And instead of one pass/fail, you score several dimensions at once:

  • Trajectory correctness — did it call the right tools, in a sensible order, with sane arguments? Sane compared to what the task actually needs: for "where is order 4412?", order_lookup("4412") is sane; order_lookup("all orders ever") is not — the agent is flailing, asking a precise tool a vague question.
  • Hallucination rate — did it invent facts the tools never returned? If the database said 287 and the answer says 300, that's a hallucination even though a tool was called.
  • Refusal rate — refusing can go wrong in two opposite directions, and this metric watches both. Direction one: refusing something legitimate — a customer asks "how many orders shipped last week?" and the agent says "I can't help with that." Direction two: failing to refuse something it should have — someone asks a read-only support agent to "delete all cancelled orders," and it cheerfully tries. Both count as refusal failures.
  • Cost and latency — dollars and seconds per task. A correct answer that takes 90 seconds and costs $2 still fails the test of real life.
Task + expected trajectory + answer check. One agent test case = the task, the trajectory you expect (tools, order, sane arguments), and a check on the final answer — scored alongside hallucination, refusal, cost, and latency.

One test case, end to end

Definitions are abstract, so let's walk one real test case all the way through, using Zentara Logistics — the made-up shipping company we've followed since Phase 2. Zentara's support agent has two tools: the text-to-SQL tool from Phase 5 (it turns a plain-English question into a database query) and a web search tool. Here's the test case:

Now the eval runner gives the agent that task and records everything it does. On a good run, the recorded trajectory looks like this:

TASK      "How many orders shipped from the Rotterdam warehouse last week?"
TOOL CALL text_to_sql("orders shipped from Rotterdam, June 22-28")   <- right tool, sane args
RESULT    287
ANSWER    "287 orders shipped from the Rotterdam warehouse last week."
COST      $0.021        TIME 4.2s

CHECKS    right tool? yes   forbidden tool avoided? yes
          answer matches tool result (287 = 287)? yes
          under budget? yes                              -> PASS

And here's a failing run — the kind that slips right past a Phase 3 style answer-only check:

TASK      "How many orders shipped from the Rotterdam warehouse last week?"
TOOL CALL (none)
ANSWER    "Around 300 orders shipped from Rotterdam last week."
COST      $0.008        TIME 1.9s

CHECKS    right tool? NO - it never touched the database
          answer matches tool result? nothing to match against
                                                          -> FAIL

Notice: the bad run is cheaper and faster, and "around 300" sounds perfectly plausible. Only the trajectory check — it never called the SQL tool — reveals that the agent made the number up. That's the entire case for grading the journey.

Plausible is the enemy. Agents rarely fail loudly. They fail by producing confident, plausible answers via a broken trajectory. If you only read the answers, you will believe your agent works long after it has stopped working.

Put it in CI

A suite you run by hand is a suite you'll forget to run. The fix is CI — "continuous integration," which sounds grand but is just this: a robot that re-runs all your checks automatically, every single time the code changes. You (well, your coding assistant) change a prompt, swap a model, tweak a tool — the robot notices, runs every test case in your suite, and reports the score.

When people say "the build fails," here's what that means in practice — and what it doesn't. It does not mean your running app breaks or users see errors. It means the robot blocks the new change from going in until the failing checks are fixed. The version people are already using keeps running untouched. GitHub Actions — a free CI robot built into GitHub, the site where your code lives — is the common choice. Wire your agent evals into it so the change is blocked if the pass-rate drops below a threshold, say 90%. Now a prompt tweak, a model swap, or a tool change can't quietly break behavior.

The dataset matters more than the tool. Promptfoo, LangSmith, Phoenix, and Braintrust all run agent evals. But the hardest, most valuable part is the same as Phase 3: writing the cases. A solid 30-case golden dataset — your hand-checked set of tasks with known-good trajectories — beats any fancy framework with three lazy tests.

Build the suite

Time to build it for real: a behavior test suite for your agent, gated in CI — your regression net (regression — when something that used to work quietly breaks) for everything that comes after. One metric in the list below deserves a quick gloss before you paste: p95 latency, which you met in Phase 6, is the waiting time your slowest runs experience — 95% of runs finish faster than this number, 5% slower. Averages hide your worst moments; p95 shows them.

Read what it builds — each case describes the behavior you expect, not just the words. This is the Rotterdam walkthrough from earlier, written down as a real test case:

# one agent test case — checks the whole trajectory, not just the final text
{
  "task": "How many orders shipped last week?",
  "expect_tools": ["text_to_sql"],     # must use the SQL tool from Phase 5
  "forbid_tools": ["web_search"],      # must NOT go to the web for internal data
  "answer_assert": "contains the number the tool returned",
  "must_not": "invent a number the database never returned",
}

# scored across the whole suite:
#   trajectory correctness · hallucination rate · refusal rate
#   $/task · p95 latency (the time your slowest 5% of runs take — Phase 6)
Deliverable — agent_evals/. A regression suite that runs in CI and catches behavior drift across prompt, model, and tool changes. From here on, you can change things without crossing your fingers.