Defining success metrics before you build

Why this matters #

If you define success after launch, you’ll define it as whatever happened. Usage went up? Success. Usage went down but power users love it? Also success. Without a metric chosen in advance, every outcome can be told as a win, which means you learn nothing.

AI features make this trickier, because “the feature was used” and “the feature helped” can diverge sharply. People click an AI button out of curiosity, get a poor result, and quietly go back to doing it by hand. This article shows how to pick metrics that catch the difference, and how to compute them from the events your feature emits.

Four layers of metrics #

Metrics for an AI feature sit in four layers. Each answers a different question, and you need at least one from each.

• Output quality is measured with evals before launch and on samples after. It tells you whether the AI is producing good output.
• User behavior tells you whether people find the output useful: do they accept it, edit it, retry, or give up?
• Business outcome tells you whether the feature matters to the business. It moves slowly and is affected by many things, so you rarely use it alone to judge a feature.
• Cost turns everything else into a decision: a feature that helps but costs more than it’s worth isn’t finished.

The layers are linked. Better quality should improve behavior, and better behavior should move outcomes. Checking that those links actually hold is the subject of linking eval scores to business KPIs.

A north star and guardrails #

From those layers, pick:

• One primary metric (sometimes called a north star for the feature): the number you’re trying to move. It should be a behavior metric close to the job, such as “share of status updates sent using the draft.”
• Two or three guardrails: numbers that must not get worse. Typical guardrails for AI features are quality complaints, the time the overall task takes, and cost per success.

Why one primary metric? Because when you have five, you’ll always find one that went up. A single primary metric, chosen before launch, makes the result unambiguous.

Why guardrails? Because primary metrics can be gamed by accident. A feature that inserts drafts automatically will raise “drafts used” while annoying everyone. A guardrail on edit rate or complaints catches that.

Metrics specific to AI features #

Standard product metrics (adoption, retention, task completion) still apply. AI features add a few that are especially informative:

Cost per success deserves special attention. Cost per request hides the true cost, because failed and retried requests cost money too. If a request costs two cents but only half of them lead to an accepted output, each success costs four cents. That’s the number to compare against the value of a success.

Exact formulas and the events each metric needs are in the metric definitions reference page.

Computing metrics from events #

Metrics are only as good as the events behind them. If your feature records a few well-chosen events, every metric above is a simple calculation. Here’s a minimal event type and the code that turns a list of events into the metric set.

/** One event per thing that happens to an AI output. Recorded by the feature, stored by your analytics tool. */
export type AiEvent =
  | { type: 'shown'; outputId: string; costUsd: number }
  | { type: 'accepted'; outputId: string; edited: boolean }
  | { type: 'retried'; outputId: string }
  | { type: 'dismissed'; outputId: string };

export interface AiMetrics {
  outputsShown: number;
  acceptanceRate: number;
  editRate: number;
  retryRate: number;
  abandonmentRate: number;
  costPerSuccessUsd: number | null;
}

/** Rate helper that returns 0 instead of NaN when there's nothing to divide by. */
const rate = (part: number, whole: number) => (whole === 0 ? 0 : part / whole);

export function computeMetrics(events: AiEvent[]): AiMetrics {
  const shown = new Map<string, number>();
  const accepted = new Map<string, boolean>();
  const retried = new Set<string>();
  const dismissed = new Set<string>();

  for (const e of events) {
    switch (e.type) {
      case 'shown':
        shown.set(e.outputId, e.costUsd);
        break;
      case 'accepted':
        accepted.set(e.outputId, e.edited);
        break;
      case 'retried':
        retried.add(e.outputId);
        break;
      case 'dismissed':
        dismissed.add(e.outputId);
        break;
    }
  }

  // Every output costs money, including the ones nobody used: that's what makes this cost per *success*.
  const totalCost = [...shown.values()].reduce((sum, cost) => sum + cost, 0);
  const edited = [...accepted.values()].filter(Boolean).length;
  // Abandoned: shown, then neither accepted nor retried (dismissed explicitly or simply ignored).
  const abandoned = [...shown.keys()].filter((id) => !accepted.has(id) && !retried.has(id)).length;

  return {
    outputsShown: shown.size,
    acceptanceRate: rate(accepted.size, shown.size),
    editRate: rate(edited, accepted.size),
    retryRate: rate(retried.size, shown.size),
    abandonmentRate: rate(abandoned, shown.size),
    costPerSuccessUsd: accepted.size === 0 ? null : totalCost / accepted.size,
  };
}

// Example: three outputs shown, one accepted unchanged, one accepted after edits, one retried.
const example = computeMetrics([
  { type: 'shown', outputId: 'a', costUsd: 0.02 },
  { type: 'shown', outputId: 'b', costUsd: 0.02 },
  { type: 'shown', outputId: 'c', costUsd: 0.02 },
  { type: 'accepted', outputId: 'a', edited: false },
  { type: 'accepted', outputId: 'b', edited: true },
  { type: 'retried', outputId: 'c' },
]);
console.log(example);
// acceptanceRate ≈ 0.67, editRate 0.5, retryRate ≈ 0.33, abandonmentRate 0, costPerSuccessUsd 0.03

Two details make this robust:

• Cost is attached to shown, not accepted. That’s what makes failures count towards cost per success.
• Rates never divide by zero. On launch day, with few events, NaN on a dashboard causes more confusion than a zero.

In production you’ll compute these in your analytics tool or a SQL query rather than in application code, but the definitions are the same. Write them down once, in code or in the metric definitions, so everyone computes them the same way. Product analytics events for AI features covers the full event schema, including model and prompt version.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #