The AI product lifecycle, from problem to iteration

Why this matters #

AI features tend to fail in predictable places. Some solve a problem nobody has. Some look great in a demo and fall apart on real inputs. Some launch to everyone at once and can’t be rolled back. Some quietly get worse after a model upgrade.

A lifecycle gives you a checkpoint before each of those failures. This article walks through the six-step loop the rest of the wiki is organized around. For each step you’ll see the question it answers, the artifact it produces and how you know you’re done.

The loop #

The lifecycle is a loop, not a line. Shipping isn’t the end: what you learn from real users becomes the next problem to solve.

Step by step #

1. User problem #

Start from a job people already do and find painful, not from a model capability. “Users spend 15 minutes catching up on long threads” is a problem. “We should add an AI summary” is a solution looking for one.

The artifact is a one-page spec: the user, the job, how they do it today, what success looks like as a number, and what you’d build without AI. That last line matters. If a filter, a template or better defaults solves the problem, ship that instead. When not to use AI covers how to decide.

2. Model and cost #

Before designing anything, check feasibility. Can current models do the task at the quality you need? How long will a response take? What will each successful use cost?

You don’t need precise numbers yet, just the right order of magnitude. If a back-of-the-envelope estimate says each summary costs more than the user is worth, find out now, not after launch.

3. AI UX #

Design the experience for the way AI actually behaves: sometimes slow, sometimes wrong, occasionally declining a request. The core question is how people will trust, check and correct the output.

The artifact covers every state: loading, streaming, success, partial answer, refusal, error and empty. If you’ve seen the code in the first article, this is where each of those branches gets its design.

4. Prototype #

Build the cheapest thing that lets real users try the experience on their real inputs. Sometimes that’s a prompt in a playground. Sometimes it’s a Wizard-of-Oz test with a human behind the curtain. Sometimes it’s a one-day Next.js app.

You’re done when you know why it helped or didn’t, not when the code is clean. Notes from watching five people use it are worth more than a polished demo.

5. Evals #

Turn what you learned into a golden dataset: real inputs paired with what a good output looks like, plus graders that check each one. Then agree on a pass threshold before you run it. Setting the bar afterwards means setting it wherever the results landed.

This is the step most often skipped, and the one that makes every later change safe. With an eval set, “I improved the prompt” becomes a number you can check.

6. Ship and learn #

Roll out behind a feature flag, to a small group first. Watch product metrics (task success, edits, retries), quality signals and cost per success. Expand if they hold. Roll back if they don’t.

Then the loop restarts. The edits users make, the questions they retry and the cases where the feature failed become new problems for step 1 and new eval cases for step 5.

What makes this loop different #

If you’ve shipped software before, most of these steps look familiar. Four things are specific to AI features:

• Evals sit between prototype and launch as a gate. A normal feature passes or fails its tests. An AI feature passes a quality threshold on a representative set.
• The loop keeps running after launch. Real usage keeps producing new failure cases, so the eval set grows over the feature’s life.
• Model changes re-enter the loop. A new model version is a change like any other: run the evals before switching.
• Cost is a per-use metric. It belongs on the launch dashboard next to quality, not in a quarterly finance review.

An eval gate in code #

Here’s step 5 in miniature: a script that runs golden cases through the summary feature, grades each result with simple checks, and fails the CI job if the pass rate drops below the threshold.

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

interface GoldenCase {
  id: string;
  thread: string;
  /** Facts a good summary must mention. Taken from real threads that users found hard to follow. */
  mustMention: string[];
}

const cases: GoldenCase[] = [
  {
    id: 'launch-date-moved',
    thread:
      'Ana: Can we still launch on the 12th?\nBen: Legal needs another week.\nAna: OK, moving launch to the 19th. Ben, tell support.',
    mustMention: ['19th', 'legal'],
  },
  {
    id: 'budget-approval',
    thread:
      'Kim: The vendor quote is 18k.\nLee: Over our 15k limit, needs director sign-off.\nKim: Asking Dana to approve by Friday.',
    mustMention: ['18k', 'Dana', 'Friday'],
  },
];

const MAX_BULLETS = 5;
// Agreed with the team before the first run, so the bar can't drift to wherever results land.
const PASS_THRESHOLD = 0.9;

/** Returns the summary, or null if the answer didn't finish normally. */
async function summarize(thread: string): Promise<string | null> {
  // No fallback here on purpose: an eval should measure the model you ship, and a refusal counts as a failure.
  const response = await client.messages.create({
    model: 'claude-opus-5-5',
    // Thinking is always on for this model and counts towards max_tokens, so leave headroom beyond the summary.
    max_tokens: 4096,
    output_config: { effort: 'low' },
    system: 'Summarize the email thread in at most five short bullet points. Name the decision that is needed, if any.',
    messages: [{ role: 'user', content: thread }],
  });

  // Only a complete answer can pass. Truncated (max_tokens), refused or otherwise unfinished answers fail the case.
  if (response.stop_reason !== 'end_turn') return null;

  return response.content.flatMap((block) => (block.type === 'text' ? [block.text] : [])).join('');
}

/** Lists the reasons a summary fails a case. An empty list means it passed. */
function grade(summary: string | null, c: GoldenCase): string[] {
  if (summary === null) return ['did not finish normally'];
  const failures: string[] = [];
  const bullets = summary.split('\n').filter((line) => /^\s*[-*•]/.test(line));
  if (bullets.length === 0 || bullets.length > MAX_BULLETS) {
    failures.push(`expected 1–${MAX_BULLETS} bullets, got ${bullets.length}`);
  }
  for (const fact of c.mustMention) {
    if (!summary.toLowerCase().includes(fact.toLowerCase())) failures.push(`missing "${fact}"`);
  }
  return failures;
}

const results = await Promise.all(
  cases.map(async (c) => ({ id: c.id, failures: grade(await summarize(c.thread), c) })),
);

for (const r of results) {
  console.log(r.failures.length === 0 ? `PASS ${r.id}` : `FAIL ${r.id}: ${r.failures.join('; ')}`);
}

const passRate = results.filter((r) => r.failures.length === 0).length / results.length;
console.log(`Pass rate ${(passRate * 100).toFixed(0)}% (threshold ${PASS_THRESHOLD * 100}%)`);

// A non-zero exit code fails the CI job, which blocks the merge.
if (passRate < PASS_THRESHOLD) process.exitCode = 1;

Run it with npx tsx evals/summary-gate.ts and an ANTHROPIC_API_KEY in your environment. Three things to notice:

• The threshold is a constant decided in advance, with a comment saying so.
• Unfinished answers fail. The script checks stop_reason before reading the text, so a truncated or refused summary can’t sneak through as a pass.
• Two cases are far too few. A real eval set starts at a few dozen cases drawn from real usage and grows from there. Simple string checks also miss a lot: Evals as Product Metrics adds rubrics and LLM-as-judge graders.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #