Capturing feedback: thumbs, edits and retries

Why this matters #

After launch, user feedback is the raw material for every improvement: new eval cases, prompt changes, UX fixes, decisions about scope. Without it, you’re optimizing against your own guesses.

The challenge is that the most obvious kind of feedback, a thumbs-up or thumbs-down button, is the one users give least. The richest signals are implicit: what people do with the output. Did they keep it, change it, regenerate it, or ignore it? This article covers both kinds, how to measure edits, and how to turn a stream of signals into concrete improvements.

Explicit feedback #

Explicit feedback is anything users tell you directly: thumbs, star ratings, “report a problem”, a reason picker. It’s valuable because it’s unambiguous about sentiment, and limited because few users give it, and those who do aren’t representative: people tend to click when something is notably good or notably bad.

To get more, and more useful, explicit feedback:

• Make it one click, right next to the output. Thumbs on every response, not a survey after the session.
• Ask why only after a thumbs-down, with a short list: “Wrong facts”, “Missed the point”, “Wrong tone”, “Too long”, “Other”. A list is faster to answer and easier to analyze than a free-text box, and invites fewer personal details.
• Don’t interrupt. Feedback prompts that block the workflow get dismissed and annoy.
• Close the loop visibly. “Thanks, this helps us improve drafts” is fine. Occasionally telling users what changed because of their feedback is better.

Implicit feedback #

Implicit feedback is what users do. Every user generates it, whether or not they’d ever click a thumb:

Each signal is noisy on its own: a retry might mean a bad answer or just curiosity; a copy might go into a document or into the bin. Combined and counted across many outputs, they’re reliable enough to compare prompt versions and spot problem categories.

Measuring edits #

Edits are the most informative signal because they show exactly what the user wanted instead. Two levels of measurement:

• How much was edited: an edit ratio from 0 (accepted unchanged) to 1 (rewritten). The editable outputs article has a word-level implementation.
• What was edited: the actual differences. “Users keep deleting the first sentence” or “users keep adding the order number” are directly actionable.

Set a threshold for a “heavy edit”, such as more than half the words changed, and treat heavily edited outputs like soft failures: they count as accepted in the funnel, but they’re candidates for review and for your eval set.

Turning signals into numbers #

The useful unit of analysis is a single output and everything that happened to it. This function takes the events for each output and classifies it, then summarizes the signals per prompt version so you can compare versions on more than acceptance alone:

type UiEvent =
  | { type: 'shown'; at: number }
  | { type: 'accepted'; at: number; editRatio: number }
  | { type: 'retried'; at: number }
  | { type: 'dismissed'; at: number }
  | { type: 'thumb'; at: number; value: 'up' | 'down' };

interface Output {
  id: string;
  promptVersion: string;
  events: UiEvent[];
}

const QUICK_RETRY_MS = 15_000; // Regenerated within 15 seconds: the first answer clearly missed.
const HEAVY_EDIT = 0.5; // More than half the words changed: accepted, but a soft failure.

type Signal = 'kept' | 'lightly-edited' | 'heavily-edited' | 'quick-retry' | 'retried' | 'dismissed' | 'ignored';

function classify(output: Output): Signal {
  const shown = output.events.find((e) => e.type === 'shown');
  for (const e of output.events) {
    if (e.type === 'accepted') return e.editRatio === 0 ? 'kept' : e.editRatio > HEAVY_EDIT ? 'heavily-edited' : 'lightly-edited';
    if (e.type === 'retried') return shown && e.at - shown.at < QUICK_RETRY_MS ? 'quick-retry' : 'retried';
    if (e.type === 'dismissed') return 'dismissed';
  }
  return 'ignored';
}

export function summarize(outputs: Output[]) {
  const byVersion = Map.groupBy(outputs, (o) => o.promptVersion);
  for (const [version, list] of byVersion) {
    const counts = new Map<Signal, number>();
    for (const o of list) {
      const signal = classify(o);
      counts.set(signal, (counts.get(signal) ?? 0) + 1);
    }
    const thumbs = list.flatMap((o) => o.events.filter((e) => e.type === 'thumb'));
    const down = thumbs.filter((t) => t.type === 'thumb' && t.value === 'down').length;
    const share = (s: Signal) => `${(((counts.get(s) ?? 0) / list.length) * 100).toFixed(0)}%`;
    console.log(
      `${version}: kept ${share('kept')}, heavy edits ${share('heavily-edited')}, quick retries ${share('quick-retry')}, ` +
        `ignored ${share('ignored')}; thumbs-down ${down} of ${thumbs.length} ratings`,
    );
  }
}

summarize([
  { id: 'a', promptVersion: 'v3', events: [{ type: 'shown', at: 0 }, { type: 'retried', at: 4_000 }] },
  { id: 'b', promptVersion: 'v3', events: [{ type: 'shown', at: 0 }, { type: 'accepted', at: 9_000, editRatio: 0.7 }] },
  { id: 'c', promptVersion: 'v4', events: [{ type: 'shown', at: 0 }, { type: 'accepted', at: 6_000, editRatio: 0 }, { type: 'thumb', at: 7_000, value: 'up' }] },
  { id: 'd', promptVersion: 'v4', events: [{ type: 'shown', at: 0 }, { type: 'accepted', at: 8_000, editRatio: 0.1 }] },
]);

The thresholds (15 seconds, half the words) are starting points: look at real outputs near each threshold and adjust until the categories match your judgment. The events come from your analytics schema, with timestamps.

From feedback to action #

Signals only matter if they lead somewhere. A simple weekly routine:

1. Triage. Pull the outputs with thumbs-down, heavy edits and quick retries from the past week. Read a sample of each, with the traces that show inputs and outputs.
2. Tag the cause. Wrong facts, missed intent, wrong tone, too long, missing context, out of scope. A handful of tags is enough.
3. Count. Which tags are growing, and which categories of requests do they cluster in?
4. Turn the top pattern into eval cases, with criteria describing what the user wanted, so the fix can be measured. See golden datasets.
5. Fix and verify through the iteration loop.

Respect privacy throughout: reviewing content means accessing users’ text, so limit who can, log access, and follow your retention rules.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #