If you have built an agentic system on your own, I have a question for you. How good was the application? Whether your answer is good or bad, here comes the next question: how did you come to that conclusion?

You built the thing. It retrieves documents, it reasons, it calls a tool when it needs to, and it gives answers that mostly look right. You demo it, people nod, and then someone like me asks: how good is it, actually?

If you don’t have an answer or your answer is something like ‘you’ve tried maybe thirty questions yourself, they seemed fine, and that’s the extent of it’, then this post is for you.

We’re going to build an eval pipeline from nothing, using one made-up application as our running example, and by the end you’ll understand what the words mean and what you’d actually do to have a solid answer next time.

PS: No prior evaluation experience assumed.

Illustration of evaluating an AI agent

Image Generated by AI

What is an eval pipeline?

Strip away the jargon and an eval pipeline is three things:

  1. A set of test questions you’ve collected, where you know what a good answer looks like.
  2. A set of checks you run against your system’s output for those questions.
  3. A habit of running it every time you change something.

That’s it. It is, roughly, unit tests for a system that doesn’t behave the same way twice.

The word “pipeline” is doing a bit of work there, and it’s worth unpacking. It’s a pipeline rather than a single test because your agent has several places it can go wrong, and you want checks that tell you which one went wrong. That’s the whole idea. Hold onto it.

The example app: a support assistant

Let’s make this concrete. Picture an internal assistant at a payments company. Support staff use it while they’re on a call with a customer.

A typical question: “This merchant says their payout last month was smaller than expected. Are they on the old fee plan, and does the refund from the 3rd explain the difference?”

To answer, the assistant can look in three places:

The assistant decides which of these to check, can check more than one, can go back and search again if what it found wasn’t enough, and has to show its sources. If it can’t find solid evidence, it’s supposed to hand off to a human rather than guess.

That last rule matters a lot, and we’ll come back to it.

Why a single score won’t work

Here’s the trap almost everyone falls into first. You write 50 questions, you check whether the final answer was right, you get 82% correct, and you feel like you’ve done evaluation.

But watch what that score hides. Suppose your assistant answers a question correctly, and along the way it searched the old tickets four times, ignored the live transactions system entirely, got lucky because a past ticket happened to contain the right number, took eleven seconds, and cited a document that doesn’t actually support what it said.

Your score says 1. You think everything is fine. But in reality, nothing is fine.

An agentic system makes decisions. It picks where to look. It decides whether to look again. It decides when to stop. Each of those is a place it can fail while still producing a right-looking answer. A single end-to-end score can’t see any of it, and worse, when your score drops from 82% to 74% after a change, it gives you no clue where to look.

So instead of one check, we build a few, each pointed at a different part of the system.

The layers, in plain terms

Your eval pipeline, if it were people: developer, curator, runner, scribe, graders, analyst, and gatekeeper on a conveyor belt

Image Generated by AI

Layer 0: Write down what happened

Before you measure anything, make your agent keep a diary. Every time it runs, it should record:

  1. the question it was asked
  2. where it decided to look
  3. what it searched for, what it found, what it answered, and what it cited

This sounds boring and it is the single most useful thing in this post. You cannot check a decision you didn’t record. Most people’s agents throw all of this away and keep only the final answer, which is exactly the piece that tells you the least.

One specific tip: record the search terms the agent made up for itself, not just the user’s original question. When retrieval goes badly, the cause is very often that the agent rewrote the question badly. It might have dropped the merchant’s ID, and searched for the general policy instead of the specific case. If you didn’t log the rewritten search, you’ll spend a day blaming your database for a mistake your agent made before it ever got there.

Layer 1: Collect your test questions

This is the unglamorous part that determines whether everything else works.

Go and find real questions people actually asked. Not questions you invented while thinking about your system. Those are always cleaner, better-phrased, and more answerable than reality. Dig through Slack, support logs, whatever you have. Take fifty, warts and all.

Then add two kinds of question that people almost always forget:

Questions your assistant should refuse. Things where the evidence genuinely isn’t there. Because remember our rule: it’s meant to escalate to a human, not guess. If your test set contains only answerable questions, you will never notice that your agent has quietly learned to always produce something. And in a payments company, a confident wrong answer told to a merchant is a far bigger problem than “I don’t know, let me check with someone.” Your tests should reflect that.

Questions that need two sources. A fee-plan rule plus a transaction fact. If every test question can be answered from one lookup, your agent’s decision-making never gets exercised, and decision-making is the part you’re trying to test.

For each question, write down the answer you’d accept, and this is the bit people skip, which documents actually support it. You’ll need that in a moment.

Fifty to a hundred questions is plenty to start. Quality of labelling beats quantity by a wide margin.

Layer 2: Check the pieces

Now, for each test question, run the agent and ask a few separate questions of the trace it left behind.

Did it find the right stuff? Compare the documents it retrieved against the documents you said were relevant. Two things can go wrong: it missed something it needed, or it dragged in a load of irrelevant material that drowned the good bit. These are usually called recall and precision, and that’s all those words mean.

Did the answer stick to what it found? Or did it add a confident detail from nowhere? This one is usually called faithfulness: is every claim in the answer actually backed by something it retrieved?

Do the citations point at the right thing? This is subtly different from the previous one and it’s the check most people miss. An answer can be perfectly grounded in the retrieved documents and still cite the wrong document for a given sentence. For our support assistant, where a staff member is going to click that citation while a customer waits on the phone, that’s its own kind of failure. Check it separately.

Layer 3: Check the decisions

This is the part that’s new for agentic systems, and it’s the reason a static RAG eval isn’t enough.

Did it look in the right place? A question about actual money movements should send it to the transactions system, not the policy PDFs. You can check this one with a plain comparison, no fancy scoring needed. Expected source, actual source, match or not.

Did it stop at the right time? Two failures, opposite directions. It stopped after one weak search and answered anyway. Or it went round and round nine times, spending money, learning nothing. Just count the steps and the cost, set a ceiling, and treat a breach as a failure.

Did it take a sensible route? Careful here, because there’s a trap. The obvious move is to write down the exact sequence of steps you expected and check the agent followed it. However, most real questions have several perfectly good routes. Checking the transaction first and the policy second is often just as valid as the reverse. If you demand an exact match, you’ll flag your agent as broken every time it’s right in a way you didn’t anticipate, and you’ll waste a week chasing regressions that aren’t real.

Check constraints instead. Did it consult the transactions system at all, for a transactions question? Did it avoid using an old support ticket as its only evidence for a policy claim? Those are the things you actually care about.

Layer 4: Score the outcome honestly

Finally, the end-to-end result which is not a single accuracy number. Sort each test into one of a few buckets:

Notice that the last two are in a different category from everything above them. That’s the point. Plain accuracy treats a wrong answer and a missed answer as equally costly, and for most real applications they are nowhere near equally costly. Decide what your bad outcomes actually cost you, and let your scoring reflect it.

A note on the tools

Ragas and DeepEval are two open-source frameworks that are widely recommended for building eval pipelines.

Use them, once you know what you want. But don’t start there. Starting with a framework tends to mean you measure whatever it measures by default, rather than what your application actually cares about, and the citation check and the escalation check, the two most valuable things in our example, are exactly the sort of thing a default metric set won’t give you.