Prompt-first prototyping

Why this matters #

Most AI features live or die on one question: can the model do the core task well enough, on real inputs? You can answer that before building any interface, database or deployment. All you need is a prompt, a folder of real examples and an afternoon.

Teams that skip this step often discover the answer weeks later, after the UI is polished and the launch is scheduled. Prompt-first prototyping moves the riskiest question to the cheapest moment. This article shows how to isolate the core task, collect inputs that tell the truth, iterate on the prompt with discipline, and run it all with a tiny harness.

The core task test #

Every AI feature has one thing the model must do well for anything else to matter. For a reply drafter, it’s writing a correct, on-tone reply from the help center. For a summarizer, it’s capturing the decisions in a thread. For a classifier, it’s picking the right category.

Write that task down in one sentence, with the input and the output:

Given a customer email and up to three help articles, write a reply that answers the question correctly using only the articles, in our friendly tone, under 120 words.

Everything else, from streaming to approval flows to analytics, can wait. If the core task doesn’t work, none of it will save the feature. If it does work, you’ll build the rest with confidence, and you’ll already have your first eval cases.

Collecting real inputs #

The single biggest factor in whether a prompt-first test tells the truth is the inputs. Invented examples are clean, short and unambiguous. Real ones are long, messy, misspelled, multilingual, angry and occasionally irrelevant.

• Use real data. Pull 20 to 30 inputs from tickets, logs, documents or interviews, with personal details removed.
• Include the awkward ones on purpose: the longest, the most ambiguous, the off-topic, the ones in another language, the ones with typos.
• Write down what good looks like for a handful of them before you run anything. It keeps you honest when you review outputs.

Twenty real inputs beat a hundred invented ones. They’re also the seed of your golden dataset.

Iterating on the prompt #

Prompt iteration goes wrong when it becomes random tweaking. A simple loop keeps it productive:

Practical habits:

• Change one thing at a time. If you change three instructions and results improve, you won’t know which one helped, or whether one of them hurt.
• Version every prompt. “v1”, “v2”, with a note on what changed. You’ll want to go back.
• Review every output, not a sample. In a 20-input test, the two worst outputs are the most informative.
• Fix the cause, not the case. If the model invents policy details, the fix is “use only the articles; say when they don’t cover it”, not a special rule for the one email where it happened.
• Know when to stop. When the remaining failures are ones the prompt can’t fix, such as missing information or an impossible input, move on to the product design.

Good prompts for product features are usually plain: what the task is, what the input is, what a good output looks like, what to do when the input is unusual, and the format to return. Clear instructions in natural language beat clever tricks.

A tiny prompt test harness #

You can run a prompt-first test in the Console workbench (opens in a new tab) by pasting inputs one at a time. For more than a handful of inputs, a script is faster and keeps every version’s results side by side. This one runs a prompt over a folder of input files and writes a Markdown review sheet with a verdict line for each.

import Anthropic from '@anthropic-ai/sdk';
import { mkdir, readdir, readFile, writeFile } from 'node:fs/promises';
import { join } from 'node:path';

const client = new Anthropic();

// Bump the version on every change so results stay comparable across rounds.
const PROMPT_VERSION = 'v1';
const SYSTEM = [
  'You draft replies to customer emails using only the help articles provided.',
  'Answer the question correctly, in a friendly tone, in under 120 words.',
  'If the articles do not answer the question, say so and suggest contacting support. Never invent policy.',
].join('\n');

const INPUT_DIR = 'prompt-tests/inputs';
const files = (await readdir(INPUT_DIR)).filter((f) => f.endsWith('.txt')).sort();
const quote = (text: string) => text.trim().split('\n').map((line) => `> ${line}`).join('\n');

const sections = [`# Prompt ${PROMPT_VERSION}`, `System prompt:\n\n${quote(SYSTEM)}`];

for (const file of files) {
  const input = await readFile(join(INPUT_DIR, file), 'utf8');
  let output: string;
  try {
    const response = await client.messages.create({
      model: 'claude-opus-5-5',
      max_tokens: 4096,
      output_config: { effort: 'low' },
      system: SYSTEM,
      messages: [{ role: 'user', content: input }],
    });
    const text = response.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('');
    // Unfinished answers are flagged in the sheet so they're reviewed as failures, not missed.
    output = response.stop_reason === 'end_turn' ? text : `⚠ Did not finish (stop_reason: ${response.stop_reason})\n\n${text}`;
  } catch (error) {
    // One failing input shouldn't stop the run; record it and carry on.
    output = `⚠ Request failed: ${error instanceof Error ? error.message : String(error)}`;
  }
  sections.push(`## ${file}\n\n**Input**\n\n${quote(input)}\n\n**Output**\n\n${output}\n\n**Verdict:** good / fixable / wrong. Reason:`);
  console.log(`done ${file}`);
}

await mkdir('prompt-tests/results', { recursive: true });
await writeFile(`prompt-tests/results/${PROMPT_VERSION}.md`, `${sections.join('\n\n')}\n`);
console.log(`Wrote prompt-tests/results/${PROMPT_VERSION}.md`);

Put one input per .txt file in prompt-tests/inputs/, run npx tsx scripts/run-prompt.ts, and open the results file. Fill in the verdict lines, count the failures by reason, change one thing in the prompt, bump the version and run again. The results folder becomes a record of what you tried and why, which is exactly what turning a prototype into a spec needs.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #