Turning a prototype into a spec

Why this matters #

A prototype teaches you things fast: which inputs break the prompt, what users actually ask for, which states confused people, what a good output looks like to them. Most of that knowledge lives in someone’s head, a chat thread and a folder of results files. Two weeks later, it’s gone.

Turning a prototype into a spec is the step that keeps it. It separates what you proved from what you hoped, records the decisions and the reasons behind them, converts prototype inputs into your first eval cases, and names the risks the full build still has to retire. This article walks through each part.

What the prototype proved #

Start with an honest list of findings, each backed by evidence. A useful discipline is to sort every claim into one of three buckets:

The point isn’t to downplay the prototype. It’s to stop “the demo went well” from being read as “the feature works”. Only the first bucket is safe to build on directly. The other two become open questions in the spec.

Decisions and their reasons #

Prototypes force decisions: which model, which prompt, what the UX does when the answer isn’t in the sources. Record each one with its reason, so nobody re-litigates it later without new evidence:

• Decision: drafts only for tickets tagged “how-to”. Because: quality on billing disputes was poor in testing (4 of 10 wrong), and those mistakes are costly.
• Decision: Claude Opus 5.5 at low effort. Because: same pass rate as medium effort on our 20 cases, faster first token.
• Decision: no draft when no article matches. Because: drafts without sources were the main reason testers rewrote replies.

The “because” is what makes this valuable. When someone proposes drafting billing replies next quarter, the record says what evidence would change the decision: a better pass rate on billing cases.

Seeding the eval set #

The inputs you tested during prototyping are real, varied and already reviewed. They’re the best possible start for a golden dataset. Don’t let them sit in a results folder.

If your prototype logged reviews as one JSON object per line, as a simple review tool or spreadsheet export can, a short script turns them into eval cases. It keeps the input, what the reviewer said a good output must do, and whether the prototype passed. It removes duplicates and keeps the failures, which are the most valuable cases.

import { readFile, writeFile } from 'node:fs/promises';

/** One reviewed prototype output, as exported from your review sheet. */
interface Review {
  input: string;
  output: string;
  verdict: 'good' | 'fixable' | 'wrong';
  reason: string;
  promptVersion: string;
}

/** A first eval case: the input, what a good answer must do, and where it came from. */
interface EvalCase {
  id: string;
  input: string;
  criteria: string[];
  /** Did the prototype pass? Failing cases are the most valuable: they show what still needs work. */
  prototypeVerdict: Review['verdict'];
  source: string;
}

const lines = (await readFile('prototype/reviews.jsonl', 'utf8')).split('\n').filter(Boolean);
const reviews = lines.map((line) => JSON.parse(line) as Review);

// Keep the latest review per input, so repeated rounds don't create duplicate cases.
const latest = new Map<string, Review>();
for (const review of reviews) latest.set(review.input.trim(), review);

const cases: EvalCase[] = [...latest.values()].map((review, i) => ({
  id: `seed-${String(i + 1).padStart(3, '0')}`,
  input: review.input,
  // The reviewer's reason becomes a first criterion; refine these by hand before relying on them.
  criteria: review.verdict === 'good' ? ['Matches the quality of the approved prototype output'] : [`Avoids: ${review.reason}`],
  prototypeVerdict: review.verdict,
  source: `prototype ${review.promptVersion}`,
}));

await writeFile('evals/seed-cases.jsonl', `${cases.map((c) => JSON.stringify(c)).join('\n')}\n`);

const failing = cases.filter((c) => c.prototypeVerdict !== 'good').length;
console.log(`Wrote ${cases.length} cases (${failing} that the prototype did not get right).`);

The generated criteria are a starting point. Before relying on the set, read each case and write criteria someone else could apply, as described in rubrics and human grading. Check that no personal data survived from the prototype logs.

Risks still open #

The last part of the spec update names what the full build must still prove, in order of how likely each is to sink the feature. Each risk gets a way to test it:

1. Quality on the full distribution of tickets. The prototype saw 20. Test: grow the eval set to 100 real tickets before launch.
2. Reply time actually drops. Assumed, not measured. Test: A/B test against the current workflow with one team.
3. Cost at full volume. Estimated from 20 requests. Test: track cost per accepted draft during the staged rollout.

The spec update #

Put it together in the one-page spec from earlier, updated with what you learned. The full flow:

A compact way to write the new sections:

## What the prototype proved (prompt test v3, Wizard-of-Oz x5, 14 Oct)
- Proved: correct drafts for how-to tickets (17/20 good, 0 harmful).
- Suggested: drafts with cited articles are trusted more (4/5 participants).
- Assumed: reply time drops (not measured yet).

## Decisions
- How-to tickets only. Because: billing disputes 4/10 wrong. Revisit if billing pass rate > 85%.
- Opus at low effort. Because: same pass rate as medium on 20 cases, faster.
- No draft without a matching article. Because: unsourced drafts were rewritten most.

## Evals
- 20 seed cases in evals/seed-cases.jsonl (5 the prototype failed). Target: 100 before launch.

## Open risks (ranked)
1. Quality on all how-to tickets → grow eval set to 100.
2. Reply time → A/B test with one team.
3. Cost at volume → track cost per accepted draft during rollout.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #