A/B testing AI features

Why this matters #

Offline evals tell you which version is better on your test cases. They can’t tell you whether users actually do better with it: whether they accept more drafts, finish tasks faster, come back more often. A version that scores higher offline can still lose in production, because it’s slower, longer-winded, or solves a problem users didn’t have.

A/B tests answer that question with real users. They’re also how you settle the trade-offs AI features constantly raise: a better model that costs twice as much, a prompt that’s more accurate but more verbose, an AI feature versus no AI feature at all. This article covers what to test, which metrics to use, how many users you need, and how to read results when cost and quality pull in different directions.

What to test #

Good candidates for A/B tests on AI features:

• The feature versus no feature. The most important test, and the one most often skipped. Does the AI feature beat the existing workflow?
• Prompt versions, once offline evals show the new one is at least as good.
• Models and effort settings: does the higher-quality, higher-cost option change behavior enough to pay for itself?
• UX changes: streaming versus not, inline versus panel, showing sources or not.

Only test variants that have already passed your offline evals. A/B tests are slow and expose real users; don’t use them to discover that a prompt is broken.

Metrics and guardrails #

Define these before the test starts, in writing:

• One primary metric: the behavior the change should improve, such as draft acceptance rate or task completion. One metric, so the result is unambiguous.
• Guardrail metrics: things that must not get worse, such as complaint rate, overall task time, or satisfaction.
• Cost per success: total model cost divided by successful outcomes, per variant. A variant that wins on the primary metric but doubles cost per success may not be worth shipping.
• The decision rule: what result leads to shipping, iterating or dropping the variant.

Use the analytics events your feature already records, tagged with the variant from your feature flag.

Sample size and duration #

The most common A/B testing mistake is stopping too early. Small samples produce big, random swings, and if you check results every day and stop as soon as one looks good, you’ll often ship a change that does nothing.

Decide the sample size before starting. It depends on your baseline rate and the smallest improvement worth detecting: small improvements need many more users. The code below includes a calculator. For example, with a 40% baseline acceptance rate, detecting a 4-point lift (to 44%) with the usual settings (5% significance, 80% power) needs about 2,400 users’ worth of outcomes per group.

Duration matters too:

• Run for full weeks, so weekday and weekend behavior are both included.
• Watch for novelty effects. New AI features often get a burst of curious usage that fades. If the effect shrinks over the test, wait for it to stabilize before deciding.
• Watch for learning effects in the other direction: some features get more useful as users learn to use them.

The statistics, in code #

This module computes the sample size needed per group, and compares two variants’ success rates with a two-proportion test, returning the difference, a 95% confidence interval and a p-value. It has no dependencies.

/** Standard normal cumulative distribution, using the Abramowitz and Stegun 7.1.26 approximation (error < 1.5e-7). */
function normalCdf(z: number): number {
  const t = 1 / (1 + (0.3275911 * Math.abs(z)) / Math.SQRT2);
  const poly = t * (0.254829592 + t * (-0.284496736 + t * (1.421413741 + t * (-1.453152027 + t * 1.061405429))));
  const erf = 1 - poly * Math.exp(-(z * z) / 2);
  return z >= 0 ? (1 + erf) / 2 : (1 - erf) / 2;
}

// Two-sided 5% significance and 80% power: the conventional defaults.
const Z_ALPHA = 1.959964;
const Z_POWER = 0.841621;

/** Outcomes needed in each group to detect an absolute lift from `baseline` with the defaults above. */
export function sampleSizePerGroup(baseline: number, minLift: number): number {
  const p1 = baseline;
  const p2 = baseline + minLift;
  const pBar = (p1 + p2) / 2;
  const n =
    (Z_ALPHA * Math.sqrt(2 * pBar * (1 - pBar)) + Z_POWER * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2 / (p2 - p1) ** 2;
  return Math.ceil(n);
}

/** Two-proportion z-test: is variant B's success rate different from A's? */
export function compareRates(successA: number, totalA: number, successB: number, totalB: number) {
  const pA = successA / totalA;
  const pB = successB / totalB;
  const pooled = (successA + successB) / (totalA + totalB);
  const z = (pB - pA) / Math.sqrt(pooled * (1 - pooled) * (1 / totalA + 1 / totalB));
  const se = Math.sqrt((pA * (1 - pA)) / totalA + (pB * (1 - pB)) / totalB);
  const diff = pB - pA;
  return { pA, pB, diff, ci95: [diff - 1.96 * se, diff + 1.96 * se] as const, pValue: 2 * (1 - normalCdf(Math.abs(z))) };
}

// Before the test: how many outcomes per group to detect 40% → 44%?
console.log('Per group:', sampleSizePerGroup(0.4, 0.04)); // 2389

// After the test: A (current prompt) vs B (new prompt), 2,450 drafts each.
const result = compareRates(980, 2450, 1078, 2450);
const pct = (x: number) => `${(x * 100).toFixed(1)}%`;
console.log(`A ${pct(result.pA)}, B ${pct(result.pB)}, lift ${pct(result.diff)}`);
console.log(`95% CI ${pct(result.ci95[0])} to ${pct(result.ci95[1])}, p = ${result.pValue.toFixed(4)}`);
// A 40.0%, B 44.0%, lift 4.0%; 95% CI 1.2% to 6.8%, p = 0.0046

// Cost per success, from the model cost each variant logged.
const costA = 2450 * 0.01; // $0.010 per draft
const costB = 2450 * 0.014; // $0.014 per draft: the new prompt is longer
console.log(`Cost per accepted draft: A $${(costA / 980).toFixed(4)}, B $${(costB / 1078).toFixed(4)}`);

For real decisions, an experimentation platform or a statistics library is a better home for this; the point of the code is to make the reasoning visible. The output shows a clear win on acceptance (the confidence interval doesn’t include zero), and it also shows that each accepted draft costs more with the new prompt. Both facts belong in the decision.

Reading the results #

AI tests often end with a trade-off rather than a clean winner. A simple way to read them:

Other situations you’ll meet:

• A flat result. No significant difference. That’s information: the change didn’t matter to users. Prefer the simpler or cheaper variant, and look for a bigger lever.
• A guardrail regression. The primary metric improved but complaints or task time got worse. Don’t ship until you understand why.
• Different results by segment. The new version helps one category and hurts another. Consider routing, but be careful: slicing results many ways will find “significant” differences by chance. Decide segments in advance.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #