Project 4: Feedback-driven feature: mine edits, improve prompts

What you’ll build #

A complete improvement loop on an AI feature that produces editable output, such as the writing assistant from Project 1. You’ll capture what users change in each output, find the patterns in those edits, turn the top pattern into eval cases, change the prompt, verify the change offline, and measure whether users actually edit less afterwards.

The deliverable isn’t a new feature: it’s a measured improvement to an existing one, with the evidence chain from user behavior to prompt change to result. That chain is exactly what hiring managers want to see.

Before you start #

• Capturing feedback: signals and the weekly triage
• Editable outputs and undo: the edit ratio function
• The iteration loop: case-by-case comparison and the change log
• Feature flags and staged rollouts: rolling the new prompt out safely

You need a feature with editable AI output and at least a few dozen real uses. Project 1 is ideal.

What you’ll practice #

• Implicit feedback: capturing edits and what changed
• Edit analysis: turning many edits into a few patterns
• Prompt iteration: one targeted change, verified
• Measuring lift: offline with evals, online with edit rates

Milestones #

1. Capture edits with diffs #

For each accepted output, record the edit ratio and, with users’ consent, the AI’s version and the final version, in a store with restricted access and short retention. Without consent, record only the edit ratio and a few structural signals (length change, sentences removed or added).

2. Cluster edit patterns #

Review the 30 most heavily edited outputs. For each, write one line on what the user changed: “removed the opening apology”, “added the order number”, “shortened from five paragraphs to two”. Group them and count. You can use a model to propose groupings, as in problem discovery, but verify every group against the actual edits.

3. Turn patterns into eval cases #

For the top pattern, write at least five eval cases from real inputs, with criteria describing what users wanted (“no apology unless the customer reports a problem”). Add them to your eval set and confirm that the current prompt fails most of them.

4. Change the prompt and run evals #

Make one targeted prompt change. Run the full eval set and compare case by case with the comparison function: the new cases should pass, and nothing that passed before should fail. Iterate until both hold, then record the change in the prompt change log.

5. Measure the lift #

Roll the new prompt out behind a flag, ideally to half of users as an A/B test, otherwise as a before-and-after comparison over equal periods. Compare average edit ratio, the share of heavy edits, and the frequency of the specific pattern you targeted. Report the result honestly, including if it’s small or unclear.

Done when #

• Edits are captured with consent and appropriate retention, and edit ratios are recorded for every accepted output.
• You have a tagged sample of heavy edits with counted patterns.
• The top pattern is covered by eval cases that failed before the change and pass after, with no regressions.
• The new prompt was rolled out behind a flag, with a measured change in edit behavior and a change log entry.

Write it up #

Tell the story as an evidence chain: what users did (edit patterns, with counts), what you changed (one prompt change, with the reason), how you verified it (eval results, case by case) and what happened (edit metrics before and after, with how you measured). Be explicit about whether the online result is from a controlled test or a before-and-after comparison.

Stretch goals #

• Automate the weekly report: heavy-edit samples, pattern counts and metrics by prompt version.
• Run two iterations and show the trend over both.
• Compare edit-based quality with thumbs-based quality and discuss where they disagree.