The iteration loop: from feedback to prompt changes

Why this matters #

Launch is where learning starts. Real users find failure modes no eval anticipated, ask for things the spec didn’t cover, and use the feature in ways that change what “good” means. An AI feature that isn’t iterated on doesn’t stay the same: as users and inputs drift, it slowly gets worse.

The teams that keep AI features improving don’t rely on bursts of heroic prompt tweaking. They run a steady loop: find the most common failure, fix it at the right layer, prove the fix with evals, ship it gradually, record what happened. This article puts together everything from this stage into that loop.

The loop #

A cycle can take a day or two. Running one per week, consistently, compounds into large improvements over a quarter.

Finding patterns in failures #

Start from the signals you already collect:

• Feedback: heavy edits, quick retries and thumbs-down with reasons, from capturing feedback.
• Traces: errors, refusals, truncations and surprising inputs, from tracing.
• Support tickets and user comments about the feature.

Read a sample, tag each failure with a cause, and count. Pick the most common pattern that matters to users, not the most interesting one. “Drafts include an apology even when the customer didn’t report a problem” in 12% of heavy edits beats a fascinating edge case seen twice.

Add eval cases first #

Before changing anything, turn the pattern into eval cases: real examples of the failure, with criteria describing what the user wanted. Then run the current version against them and confirm they fail.

This order matters for three reasons:

• It proves you understand the failure. If you can’t write a case that reproduces it, you don’t understand it yet.
• It makes the fix measurable. “Fixed 9 of 10 apology cases” is a result; “the prompt feels better” isn’t.
• It prevents the failure from returning. The cases stay in the golden dataset and the CI suite.

Choosing the fix #

Prompt changes are the obvious lever, but not always the right one. Match the fix to the cause:

Prefer general fixes over special cases. A prompt that grows a new “if the customer mentions X, do Y” rule for every failure becomes brittle and contradictory. If you find yourself writing special cases, step back and ask what general principle they share.

Verifying before shipping #

A change is ready when two things are true: the new cases pass, and nothing that used to pass now fails. Overall pass rates can hide the second. A change that fixes five cases and breaks five others has the same score as no change at all.

Compare runs case by case:

/** One eval run: which cases passed. Produced by your suite runner for each prompt version. */
type Run = Record<string, boolean>;

/** Case-by-case comparison of two runs: what got fixed, what broke, and the net change. */
export function compareRuns(before: Run, after: Run) {
  const ids = [...new Set([...Object.keys(before), ...Object.keys(after)])].sort();
  const fixed = ids.filter((id) => before[id] === false && after[id] === true);
  const broken = ids.filter((id) => before[id] === true && after[id] === false);
  const added = ids.filter((id) => !(id in before));
  const passRate = (run: Run) => Object.values(run).filter(Boolean).length / Object.values(run).length;
  return { fixed, broken, added, before: passRate(before), after: passRate(after) };
}

const v4: Run = { 'apology-1': false, 'apology-2': false, 'refund-1': true, 'export-1': true, 'tone-1': true };
const v5: Run = { 'apology-1': true, 'apology-2': true, 'refund-1': true, 'export-1': false, 'tone-1': true, 'apology-3': true };

const diff = compareRuns(v4, v5);
console.log(`Pass rate ${(diff.before * 100).toFixed(0)}% → ${(diff.after * 100).toFixed(0)}%`);
console.log(`Fixed: ${diff.fixed.join(', ') || 'none'}`);
console.log(`Broken: ${diff.broken.join(', ') || 'none'}`); // export-1: investigate before shipping
console.log(`New cases: ${diff.added.join(', ') || 'none'}`);

The overall pass rate went up, but export-1 broke. Read that output before shipping: the fix for apologies might have shortened replies enough to drop the export instructions. Either refine the change until nothing breaks, or decide explicitly that the trade-off is worth it and record why.

Then ship the change behind a flag, roll it out gradually, and check that the production signal that started the cycle improves: fewer heavy edits mentioning apologies, in this example.

Keeping a change log #

Record every change in one place, next to the prompts themselves. Each entry:

## reply-v5 (2026-10-17)
- Pattern: unnecessary apologies in 12% of heavily edited drafts (week of 6 Oct).
- Change: "Apologize only if the customer reports a problem or a delay."
- Evals: apology cases 0/3 → 3/3; export-1 broke and was fixed in v5.1; overall 82% → 88%.
- Rollout: 10% from 14 Oct, 100% from 17 Oct.
- Result: heavy edits mentioning apologies fell from 12% to 3% of heavy edits.

A change log turns iteration from tribal knowledge into a record. It answers “why does the prompt say this?” months later, prevents someone from reverting a deliberate change, and becomes the raw material for your case studies.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #