Project 6: Capstone: a paid AI SaaS feature end to end

What you’ll build #

One AI feature that real people pay for, taken through every step of the AI product lifecycle: a problem with evidence, a one-page spec, an experience with every state designed, an eval suite gating changes in CI, a staged rollout, a pricing model grounded in unit economics, a metrics and cost review, and a published case study.

“Paid” can be small: a paid tier in your own side project, a feature in a product at work (with permission), or a tool you sell to a handful of customers. The point is that someone decides the feature is worth money, which tests your judgment far harder than a demo.

Before you start #

This project uses the whole wiki. You’ll lean most on:

• Writing a one-page AI feature spec and defining success metrics
• Error, refusal and empty states and streaming
• Regression eval suites in CI
• Feature flags and staged rollouts
• Unit economics and pricing
• Writing a product case study

What you’ll practice #

Everything: product discovery, specs, AI UX, evals, rollout, pricing, measurement, and communicating the result.

Milestones #

1. Write the one-page spec #

Start from a problem with evidence, found through discovery and interviews. Write the spec with all eight sections, including the quality bar with a dated threshold, failure states, a cost estimate and the non-AI alternative. Run a prompt-first test before committing.

2. Design every state of the UX #

Design the full experience: entry point and onboarding, streaming or loading, the result with edit controls and sources where relevant, and every unhappy state from the typed state table. If the feature takes actions, add approvals. Force every state before launch.

3. Build the eval suite and CI gate #

Build a golden dataset of at least 50 real cases with categories and must-never-fail cases, graders from code checks up to a calibrated judge where needed, and a CI gate on prompt and model changes. Add guardrails for the failures that must be impossible.

4. Roll out behind flags, with pricing #

Price the feature from your unit economics: cost per success as the floor, value as the ceiling, limits that cover heavy users. Launch behind a flag in stages with exit criteria, tracing, analytics events, rate limits and budgets, and a tested kill switch.

5. Run a metrics and cost review #

After at least two weeks with paying users, review: primary metric and guardrails, eval results on recent production samples, cost per success and margin by segment, feedback patterns and incidents. Run at least one iteration loop on the top failure pattern, with a change log entry.

6. Write and publish the case study #

Write the case study: summary, problem with evidence, approach, two or three key decisions with rejected options, results (evals, metrics, cost, revenue), and lessons. Publish it in your portfolio.

Done when #

• At least one person has paid for the feature.
• The spec, eval set, CI gate, flag configuration, dashboards and change log all exist and are linked from the case study.
• Every UX state is designed and was forced before launch; guardrails cover the must-never-fail cases.
• Pricing is grounded in measured cost per success, with limits for heavy use.
• A metrics and cost review was held, at least one improvement shipped through the iteration loop, and the case study is published.

Write it up #

This case study is the centerpiece of your portfolio. Lead with the outcome in one paragraph: who pays, for what, and what changed for them. Then focus on the decisions that only an AI product engineer would make: what you chose not to automate, how you defined quality, how you priced against cost, and what the first iteration taught you. Include the numbers, how you measured them, and an honest lessons section.

Stretch goals #

• Run an A/B test on a pricing or quality change, as in Project 5.
• Add a model upgrade process: evaluate a newer model against your suite and migrate behind a flag.
• Present the case study as a 10-minute talk to a meetup or your team, and incorporate the questions you get.