The struggle

The feature works. You’ve used it. Your engineer has used it. Someone in the launch review asks the reasonable question — is it good enough? — and what you have is a feeling assembled from eleven ad-hoc tries, most of them on inputs you invented, none of them written down.

So the decision gets made on confidence rather than evidence, and then the real problem starts: three weeks later someone changes the prompt or the model version moves under you, and nobody can tell whether it got better or worse. There’s nothing to compare against. The obvious fix — “let’s just test it more carefully” — fails for a specific reason: careful manual testing doesn’t survive contact with a second release. It’s expensive, it’s a different person each time, and its output is an opinion, not a baseline.

The thing that fixes it isn’t more testing. It’s an eval set: real inputs, an explicit view of what a good answer looks like, graded automatically, re-runnable in a minute. It’s a product document that happens to execute — and writing it needs someone who knows what customers actually need, which is why it lands on you rather than on engineering.

The chain

Step 1 — Gather real inputs, not imagined ones (30 minutes)

Pull 40 inputs your feature has genuinely seen: support tickets it summarised, queries it answered, documents it classified. Export from the logs, the support tool, wherever they live.

Forty is a deliberate number and it’s the first place people go wrong. The instinct is to hand-craft eight perfect test cases; Anthropic’s eval guidance says the opposite in as many words — “more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” Volume is what catches the failure you didn’t predict. Eight cases test your imagination.

Deliberately include the ugly ones: empty input, input three times longer than normal, input in the wrong language, something abusive, something ambiguous where two answers are defensible. Those are the categories that break in production and never appear in a hand-picked set.

Step 2 — Draft the cases and the rubric · Claude

Paste the inputs into Claude and get the first pass of your grading criteria. You are not asking it what good looks like — you’re asking it to turn what you already know into something checkable, and to tell you where your own definition is vague:

I'm building an eval set for an AI feature. Below are ~40 real inputs it receives.

The feature is: <one sentence — what it takes in, what it produces, for whom>.
A good output must: <2–4 things you actually care about, in your words>.

For each input, produce:
1. A short label for what kind of case it is (typical / edge / adversarial / ambiguous).
2. ONE specific, checkable assertion about what a good response must or must not do
   for THIS input. Prefer literal checks ("must contain the ticket ID", "must not
   name a competitor") over subjective ones. Only fall back to a judged criterion
   ("tone is professional") when nothing literal captures it.
3. If my stated criteria don't determine the right answer for this input, say so
   and ask me the question instead of guessing.

Hard rules: do not invent inputs — use only the ones I gave you. Do not write an
assertion you couldn't check mechanically. Flag every case where two different
answers would both be acceptable; those are mine to decide, not yours.

Inputs:
"""
<paste the 40 inputs>
"""

The last two rules are what make this useful rather than flattering. The cases it flags as ambiguous are the actual product decisions hiding inside your feature — the ones nobody has made yet. Make them. That list is worth the exercise on its own, even if you stop here.

Step 3 — Make it runnable · promptfoo

In an empty folder, npx promptfoo@latest init --example getting-started writes a working config you edit rather than a blank file. Put your prompt in, your 40 cases in, and start with literal contains / not-contains assertions before touching anything cleverer. Then npx promptfoo@latest eval and npx promptfoo@latest view for the side-by-side grid.

If the YAML is the wall — and for most PMs it is — this is the single best first job for Claude Code: point it at the folder and describe the cases in English. It writes the config. You still own what’s in it.

When to skip this step: if your feature ships once and never changes, a spreadsheet of 40 rows you grade by hand is genuinely fine. The tooling earns its cost on the second run, not the first.

Step 4 — Grade the subjective ones with a different model

Some criteria won’t reduce to string matching — tone, whether a summary preserved the point, whether a refusal was appropriate. For those, use LLM-graded assertions, and grade with a different model than the one generating. Anthropic’s guidance states it plainly: “generally best practice to use a different model to evaluate than the model used to generate the evaluated output.”

This is the step most teams get wrong, and the failure is silent: a model asked to grade its own output is systematically generous, so your eval passes and your customers still complain. Nothing in the output tells you this is happening.

Step 5 — Run it on every change, and report the diff

The set has done its job the first time someone changes a prompt and you can say “seven of forty regressed, here they are” instead of “it feels worse.” Re-run before each release. Put the pass rate and the flipped cases in the launch doc.

Then keep feeding it: every production complaint becomes a new case. An eval set that doesn’t grow is a snapshot of what you were worried about in July.

Where it breaks

Passing an eval set that misses the real failure mode. The most dangerous outcome here, because it reads as safety. 100% pass means you covered what you thought to cover. Add every genuine complaint as a case, and be suspicious of a set that has never gone red.
Grading with the generating model. Silently inflates every subjective score. See step 4 — this is the single most common technical mistake, and it produces confident, wrong numbers.
Hand-crafting eight beautiful cases. The intuitive move and the wrong one. Coverage beats polish; a rough case you actually included beats a perfect one you didn’t.
Letting Claude write the criteria for you. It should turn your judgment into checkable assertions, not supply the judgment. If you can’t say what good means for your customers, no tool closes that gap — and that’s the part of this job nobody can take from you.
Cost surprise on LLM-graded assertions. Every judged case is another model call, on every run. Forty cases with three judged assertions is 120 calls a run; keep literal checks literal.
Treating the number as the decision. 92% pass is not a launch approval. It’s evidence you bring to a judgment that’s still yours — and the seven failures matter more than the ninety-three passes if they’re the ones customers hit most.

“Someone asks whether the AI feature is good enough to ship, and the honest answer is that you tried it eleven times and it felt fine.”