Linking eval scores to business KPIs

Why this matters #

An eval score that nobody connects to business outcomes gets ignored. Leadership doesn’t fund “groundedness went from 88% to 94%”. It funds fewer support tickets, faster onboarding, higher conversion. If you can’t explain how an eval improvement turns into one of those, AI quality work loses every prioritization debate to features with a clearer story.

The opposite risk is just as real: an eval score can rise while the business metric doesn’t move, because the eval measures something users don’t care about. This article shows how to draw the chain from eval to KPI, check that it holds, notice when it breaks, and report quality in terms that earn a place in planning.

The chain from eval to KPI #

Every AI feature has a chain of cause and effect from output quality to business value. Writing it down makes your assumptions explicit, and testable.

Cost runs alongside the chain: each step can be undone by a cost per success that’s higher than the value it creates. That’s why cost per success belongs on the same dashboard.

Write the chain for each feature in its spec, with the metric at each step. Then the question “is the eval worth improving?” becomes “does moving the first box move the last one?”

Checking the link #

The chain is a hypothesis until data supports it. Three ways to check it, from strongest to weakest:

1. A/B test a quality change. Ship a version that scores higher offline to half of users, and compare behavior and outcomes. This is the cleanest evidence that the eval predicts real impact. A/B testing AI features covers how.
2. Compare segments. If quality varies by category (say, how-to tickets score 92%, billing 70%), check whether user behavior follows the same pattern: are billing drafts accepted less often?
3. Track both over time. Plot weekly eval scores against weekly behavior metrics and look at whether they move together.

The third is the easiest and the weakest, because many things change week to week. Here’s a small helper that computes the correlation between two weekly series, with the cautions built into the output:

/** Pearson correlation between two equal-length series. Returns null if there's no variation to compare. */
export function correlation(xs: number[], ys: number[]): number | null {
  if (xs.length !== ys.length || xs.length < 3) throw new Error('Need two series of the same length, at least 3 points.');
  const mean = (v: number[]) => v.reduce((a, b) => a + b, 0) / v.length;
  const mx = mean(xs);
  const my = mean(ys);
  let cov = 0;
  let vx = 0;
  let vy = 0;
  xs.forEach((x, i) => {
    const y = ys[i] ?? 0;
    cov += (x - mx) * (y - my);
    vx += (x - mx) ** 2;
    vy += (y - my) ** 2;
  });
  return vx === 0 || vy === 0 ? null : cov / Math.sqrt(vx * vy);
}

// Eight weeks: offline pass rate after each week's release, and the share of drafts agents accepted.
const evalPassRate = [0.81, 0.83, 0.82, 0.86, 0.88, 0.87, 0.91, 0.92];
const acceptanceRate = [0.42, 0.46, 0.41, 0.45, 0.51, 0.46, 0.5, 0.55];

const r = correlation(evalPassRate, acceptanceRate);
console.log(r === null ? 'No variation to compare.' : `Correlation over ${evalPassRate.length} weeks: ${r.toFixed(2)}`);
// A high correlation over a few weeks is encouraging, not proof: seasonality, new users or a UI change
// could move both. Confirm with an A/B test before claiming the eval drives acceptance.

A strong correlation over several weeks supports the chain; a weak one is a warning. Neither is proof: confirm important links with an experiment.

When the link breaks #

Sometimes the eval improves and nothing else moves. That’s valuable information, and it usually means one of these:

• The eval measures the wrong thing. Graders reward something users don’t value, such as thoroughness when users want brevity. Go back to interviews and real usage, and revise the criteria.
• The eval set drifted. Real requests changed (new product areas, new kinds of users), but the cases didn’t. Refresh the golden dataset from recent traffic.
• Something else is the bottleneck. Quality is good enough; users are held back by discoverability, latency or trust. More eval points won’t help; fix the experience.
• Goodhart’s law. “When a measure becomes a target, it ceases to be a good measure.” If the team optimizes prompts against the eval set directly, the score can rise by fitting those cases without generalizing. Keep a held-out set that nobody tunes against, and refresh cases regularly.

The held-out set is the simplest defense: a portion of the golden dataset used only to report scores, never to guide prompt changes. If the tuning set improves and the held-out set doesn’t, you’re overfitting.

Reporting quality to leadership #

Leadership needs one view that connects quality to outcomes, not a table of criteria. A useful monthly report for each AI feature:

• One chart with three lines over time: the eval pass rate on the held-out set, the main behavior metric (acceptance, completion), and cost per success.
• The business outcome it drives, with the latest number and the trend.
• What changed and what’s next: the changes shipped this period, their measured effect, and the next bet with its expected impact.
• Risks: must-never-fail results and any incidents.

Keep the eval details available but secondary. The message is: “quality went up, users accepted more drafts, reply time dropped, and each accepted draft costs less than before.” That sentence, backed by a chart, earns AI quality work its place in planning.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #