Project 5: A/B-tested AI feature with a cost-per-success dashboard

What you’ll build #

An AI feature with two variants that differ in a way that affects both quality and cost, for example a capable model at medium effort versus the same model at low effort, or a short prompt versus a longer one with examples. You’ll ship both behind a feature flag, assign users consistently, record analytics events, and build a dashboard showing success rate and cost per successful task for each variant. At the end you’ll write a decision memo: which variant to ship, and why.

This project is about making a product decision with data when quality and cost pull in different directions, which is the most common kind of decision in AI product work.

Before you start #

• Feature flags and staged rollouts: stable bucketing and the flag config
• A/B testing AI features: metrics, sample size and the statistics module
• Product analytics events: the event schema
• Cost dashboards and budgets: aggregation and anomaly checks

You need a feature with enough usage to reach a meaningful sample. If you don’t have real traffic, simulate it honestly with a script that runs real inputs through both variants, and say so in the write-up.

What you’ll practice #

• Feature flags with consistent assignment and logged decisions
• A/B testing with a pre-registered decision rule
• Analytics events that support quality and cost metrics per variant
• Cost dashboards showing cost per success, not just cost per request

Milestones #

1. Define variants, metrics and guardrails #

Choose two variants that have both passed your offline evals. Write down, before any data: the primary metric (for example, acceptance rate), two guardrails, how you’ll compute cost per success, the sample size from the calculator, and the decision rule (“ship B if acceptance rises at least 2 points with no guardrail regression and cost per success rises less than 30%”).

2. Implement flags and assignment #

Use the bucketing code or your flag service to assign users 50/50. The flag decision carries the prompt version and model, and every request logs which variant it used. Include a kill switch and test it.

3. Instrument analytics events #

Send the events from the analytics schema with the variant field set, including costUsd on ai_completed from the response’s usage. Verify in staging that both variants produce complete event sequences.

4. Build the cost-per-success dashboard #

Build a dashboard, in your analytics tool, a notebook or a small page, with: requests, success rate and cost per success per variant per day; total spend by variant; and the guardrail metrics. Add an anomaly check on daily spend.

5. Run the test and write the decision memo #

Run until you reach the planned sample size and at least a full week. Compute the results with the statistics module, check guardrails, and apply your decision rule. Write a one-page memo: variants, metrics, results with confidence intervals, cost per success for each, the decision, and what you’ll test next.

Done when #

• Two variants run behind a flag with stable assignment, logged decisions and a tested kill switch.
• Events record variant, outcome and cost for every request.
• A dashboard shows success rate and cost per success by variant, plus guardrails and spend.
• The test reached its planned sample size, and a decision memo applies the pre-registered rule.

Write it up #

The decision memo is the core of the case study. Add the dashboard screenshot, explain the trade-off you faced between quality and cost, and say what value per success you assumed and why. If the result was flat or surprising, say so: a well-run test with an honest “no difference” is a strong portfolio piece.

Stretch goals #

• Add a third variant with a smaller model for a sub-step, and compare all three.
• Segment results by a dimension chosen before the test, such as request type.
• Automate a weekly report emailed to yourself with the dashboard’s key numbers.