Why evals are product metrics

Why this matters #

Without evals, every change to an AI feature is a guess. You tweak a prompt, try three examples, they look better, you ship. Two weeks later a different kind of request starts failing and nobody knows which change caused it. Model upgrades are worse: a new version is better on average and different in a hundred small ways.

An eval turns “it looks better” into a number you can check. That makes evals the most important tool in AI product engineering, and it’s why they belong to the product, not to a notebook someone runs once. This article explains what an eval is, why it’s a product metric, how offline evals relate to what happens in production, and who should own them.

What an eval is #

An eval has four parts:

1. Cases: real inputs the feature will see, ideally drawn from actual usage.
2. Expectations: what a good output must do for each case. Sometimes an exact answer, more often criteria: “mentions the refund deadline”, “doesn’t promise anything the policy doesn’t allow”.
3. Graders: something that checks outputs against the expectations: code, a model acting as a judge, or a person.
4. A score: usually a pass rate, sometimes broken down by category, compared against a threshold agreed in advance.

Here’s the smallest useful version, with a deterministic grader so it runs without an API key. Real evals call the feature itself instead of the stand-in.

interface Case {
  id: string;
  category: 'how-to' | 'billing';
  input: string;
  /** Facts a good answer must include. */
  mustInclude: string[];
}

interface Result {
  id: string;
  category: Case['category'];
  passed: boolean;
  failures: string[];
}

/** Stand-in for the real feature. In a real eval this calls your prompt and model. */
async function feature(input: string): Promise<string> {
  return input.includes('export') ? 'Go to Billing > History and click Export CSV.' : 'Please contact support.';
}

function grade(output: string, c: Case): Result {
  const failures = c.mustInclude.filter((fact) => !output.toLowerCase().includes(fact.toLowerCase())).map((f) => `missing "${f}"`);
  return { id: c.id, category: c.category, passed: failures.length === 0, failures };
}

const cases: Case[] = [
  { id: 'export-1', category: 'how-to', input: 'How do I export invoices?', mustInclude: ['Billing', 'Export'] },
  { id: 'refund-1', category: 'billing', input: 'I was charged twice', mustInclude: ['refund'] },
];

const results = await Promise.all(cases.map(async (c) => grade(await feature(c.input), c)));

// Break the score down by category: an average can hide a category that fails completely.
for (const category of ['how-to', 'billing'] as const) {
  const inCategory = results.filter((r) => r.category === category);
  const passed = inCategory.filter((r) => r.passed).length;
  console.log(`${category}: ${passed}/${inCategory.length} passed`);
}
for (const r of results.filter((r) => !r.passed)) console.log(`FAIL ${r.id}: ${r.failures.join(', ')}`);

That breakdown by category is the first habit worth building. An overall pass rate of 80% can mean “80% everywhere” or “100% on easy cases and 20% on the ones users care most about”.

Evals measure user value #

What makes an eval a product metric is where the expectations come from. A useful eval defines “good” from the user’s point of view:

• Cases come from real use: the questions people actually ask, the documents they actually upload, including the messy ones. See golden datasets.
• Criteria describe what users need: correct, complete enough to act on, grounded in sources, in the right tone, honest about what it doesn’t know.
• Categories match how the product is used: by task type, customer segment, language or difficulty, so you can see where the feature serves users well and where it doesn’t.

An eval built this way answers the question product teams care about: “for the people using this feature, how often does it do the job?” That’s why the eval threshold appears in the one-page spec, and why a drop in eval score is a product incident, not an ML curiosity.

The eval pyramid #

Different graders trade cost, speed and reliability. Like a test pyramid in software, you want many cheap checks at the bottom and fewer expensive ones at the top:

• Code checks cover everything that can be checked exactly. Write as many as you can; they run on every change.
• LLM-as-judge covers qualities code can’t check, such as tone, completeness or faithfulness to sources. See LLM-as-judge.
• Human grading is the reference standard: slow and expensive, but it’s how you know the automated graders are right. See rubrics and human grading.
• Online metrics measure what actually happens with real users, the final judge of whether the offline evals predicted anything.

Offline evals and online metrics #

Offline evals run on a fixed set of cases, before a change ships. Online metrics come from real usage after it ships: acceptance rate, edit rate, retries, task completion. You need both, and they play different roles:

The two connect in a loop. Offline evals let you ship changes safely; online metrics tell you whether the eval set reflects reality; and production failures become new eval cases. When an offline improvement doesn’t move online metrics, that’s a signal your eval set has drifted from what users need, covered in linking eval scores to business KPIs.

Who owns the eval set #

Because the eval set defines what “good” means, it’s a product artifact, and it needs an owner who understands users. In practice:

• The AI product engineer owns it day to day: keeps it current, adds cases from failures, maintains graders, runs it in CI.
• Domain experts contribute criteria and grading: support leads for support replies, lawyers for contract features.
• The team agrees on the threshold before results are known, and records it.
• It’s versioned like code, so you can see when cases and criteria changed and why.

An eval set nobody owns slowly stops reflecting real use, and a stale eval set is worse than none: it gives confidence it hasn’t earned.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #