PHASE 03 · EVAL
Goal: Measure quality systematically instead of eyeballing outputs.
Evaluation is how you know your LLM app actually works — not by reading a few outputs and saying "looks good", but with a test suite that runs on every change. Without eval, you're flying blind: a small prompt tweak can quietly break 30% of cases and you'll never notice until a user complains.
Promptfoo is the most popular open-source eval framework. You write test cases in YAML (a question + assertions like "answer must contain X" or "an LLM judge gives this a 4/5"), define one or more providers (model + prompt template combinations), and run promptfoo eval. It gives you a side-by-side table with pass/fail, latency, and cost per cell.
There's a vocabulary worth learning: offline eval (running tests against a fixed dataset, like CI) vs. online eval (measuring real user interactions in production). You're starting with offline. The hardest part isn't the tool — it's writing good test cases.
npm install -g promptfoo.promptfooconfig.yaml: e.g. (Sonnet + naive chunking), (Sonnet + recursive chunking), (Haiku + recursive chunking), (Haiku + verbose prompt).equals and contains for factual questions, llm-rubric (LLM-as-judge) for fuzzy ones, javascript for custom checks.promptfoo eval and open the web UI. Pick a winner based on pass-rate × cost × latency — not just accuracy.equals, contains, and llm-rubric assertions?promptfooconfig.yaml — A results table showing which model + prompt + chunking combo wins on your data.
Concepts unlocked: offline eval · regression testing · LLM-as-judge · data-backed selection
You just built rag.py in Phase 2. How do you know it's any good? Probably the same way everyone starts: you ran a few questions, read the answers, and thought "yeah, looks fine." That feeling is exactly what this phase fixes.
Reading a handful of outputs hides two things. You only ever see the cases you happened to try — and when you tweak a prompt or swap a model, you have no way to tell whether you just made it better or quietly broke it.
The fix borrows a name from ordinary software: a test suite. Don't let the term put you off — a test case is just a question you already know the right answer to, written down together with that answer. A test suite is a saved list of those, kept in a plain file, that you can run over and over. No code needed to understand it: it's a quiz you wrote for your own app, with the answer key attached.
The whole shift. Stop asking "does this output look good?" and start asking "what percent of my test cases pass?" That number is something you can track, compare across changes, and push up over time.
An eval is that test suite idea, applied to your LLM app: a list of cases, each with a question and a check for whether the answer is acceptable. You run the whole list and get a score.
Make it concrete. For your Phase 2 RAG bot, one test case might be: the question "what's our refund window?" with the check "the answer must contain '30 days'." Write 30–50 of these — easy ones and tricky ones — run them all, and you get a single number: this version passed 44 of 50. Tweak a prompt, re-run, and you can see at a glance whether that number went up or down.
There's one thing we glossed over: what does the check actually compare the answer to? And what really runs when you hit "go" on 50 test cases? That's next.
Offline vs online. You're building offline eval — a fixed set of cases you run yourself, like a test suite. Later, online eval watches real users in production — the software word for when your app is live and real people, not just you, are using it. Start offline; it catches most breakage before it ever reaches a user.
Every test case needs two things: a question, and what the right answer actually looks like — written by a human, once, ahead of time. That whole collection of question-and-right-answer pairs has a name: your golden dataset. "Golden" just means: this is the ground truth everything else gets checked against. (If the word rings a bell — Phase 2's pitfalls section talked about checking whether the gold chunk made it into retrieval — it's the same use of the word: gold = the known-correct reference.)
A few rows from Zentara's golden dataset might look like this:
question reference answer
-------------------------------------- ------------------
what's our refund window? 30 days
do you ship internationally? No, US only
what's the parental leave policy? 18 weeks paid
who won the 2049 World Cup? (trap — not in the docs;
should say it doesn't know)You write the reference answer yourself, from what you already know is true — you never ask the model to grade its own homework. This is also, honestly, the tedious part: writing 30–50 of these well is most of the real work in eval.
Steal from reality. The best rows aren't invented at a desk. Every time someone asks your bot a question and gets a bad answer, that question — with the answer it should have given — becomes a new row. Your golden dataset grows out of real failures.
So you have a file of questions and reference answers. What actually happens when you press "go"? Let's run one row all the way through.
An eval run is the same short trip, repeated for every row of the golden dataset. Here's one full trip, on the course's running example — Zentara's leave-policy question:
Now a row that fails — and why the fail is the whole point. Question: "do you ship internationally?" Reference answer: "No, US only." Your system answers: "Yes — Zentara ships to over 40 countries worldwide." The check looks for "US only", doesn't find it → ✗ fail. That answer is a confident hallucination — Phase 2's worst pitfall — and you would almost certainly have missed it by skimming three outputs. The scoreboard just caught it for you.
Every row takes the same trip. As one picture:
That's one row at a time. Here's a whole run — Zentara test questions flowing through your RAG system, each actual answer compared against its golden answer, the scoreboard filling as verdicts land:
Final score: 3 of 4 — a 75% pass rate. And notice which part of the screen matters most: the ✗. A passing row only confirms what you hoped; the failing row names the exact question, the exact wrong answer, and the reference it missed. That's your to-do list, for free, on every run.
The whole mechanism, in one line. For every row in your golden dataset: run the question through your real system, compare the actual answer to the golden reference answer, tally pass or fail. The score you get out is your eval.
The one blank left: how does the check decide "close enough," when two correct answers can be worded completely differently? That's not one-size-fits-all — there are a few kinds of checks, from strict to fuzzy.
So how does that comparison actually decide match-or-no-match? Three kinds of checks, from strict to fuzzy — each shown on one Zentara test case:
See how the choice matters on one question — "is our office open on Saturdays?" An exact-match check that demands the answer be literally "No." is too strict: the model might say "No, we're closed on weekends," which is correct but fails. A contains check for the word "closed" is more forgiving and still catches wrong answers. And for something open-ended like "summarize our weekend hours," there's no single right wording, so only an LLM-as-judge will do.
Rule of thumb. Use the strictest check you can get away with. Exact match where a single answer exists, contains where a key fact must appear, and save the LLM judge for genuinely open-ended answers — it's the most powerful and the most expensive.
LLM-as-judge is handy, not free or perfect. Every judged test costs an extra model call, and the judge can be wrong or biased — it tends to favour longer, more confident-sounding answers. Use it only for the fuzzy cases; use exact or contains wherever a real check exists.
Time to wrap a real test suite around the rag.py you built in Phase 2. You'll use Promptfoo, the popular open-source eval tool: you describe your test cases and a few model setups in one file, and it runs them all into a side-by-side table. As always — your assistant builds it, you run it.
Read what it generates. The config is one file: a list of setups to compare, and a list of test cases, each with a check. The same test cases run through every setup, so you get a fair side-by-side:
One thing before you look at the file: it's written in YAML. This isn't code — it's a settings file, like a structured shopping list: labeled sections, indented items, nothing that executes. You can read it top to bottom like a form your assistant filled in.
# run the same test cases across several setups
providers: # a "provider" = one setup being tested (a model + your pipeline)
- file://rag_provider.py # a "wrapper" — a small connector script your
# assistant writes so promptfoo can call
# your rag.py's answer()
tests:
- vars: { question: "What is our refund window?" }
assert:
- type: contains
value: "30 days"
- vars: { question: "Summarize our onboarding policy." }
assert:
- type: llm-rubric
value: "Accurate and grounded in the retrieved docs; no invented facts."
- vars: { question: "Who won the 2049 World Cup?" } # the trap row
assert:
- type: llm-rubric
value: "Says it doesn't know, or that it's not in the documents."Deliverable — promptfooconfig.yaml. A results table that shows which model + chunking + prompt combo wins on your questions. Re-run it every time you change anything — it's your safety net for the rest of the course.