Building golden datasets from real user tasks

Why this matters #

An eval is only as good as its cases. A set of tidy, invented examples measures how the feature handles tidy, invented examples, which is not what users send. Real tasks are longer, messier, more ambiguous and more varied, and they’re where the failures that matter hide.

A golden dataset is a curated set of real inputs, each paired with what a good output must do. Building one well is mostly careful, unglamorous work: choosing where cases come from, making sure hard cases are represented, writing expectations someone else could apply, protecting privacy, and keeping the set current. This article covers each step.

Where cases come from #

The best cases are things users actually did. Common sources:

Before launch, you’ll lean on prototypes, tickets and experts. After launch, production logs and feedback become the main source, and the set improves every week. The prototype-to-spec step shows how to seed it.

Coverage over volume #

A good first golden set is often 50 to 200 cases: enough to cover the categories that matter, small enough to read and grade carefully. Coverage matters more than size:

• Common cases, in roughly their real proportions, so the overall score means something.
• Hard cases, deliberately over-represented: long inputs, ambiguous requests, multiple languages, unusual formats. A random sample contains few of these, yet they’re where most failures are.
• Known failure modes: every bug report and production failure becomes a case, so it can’t silently come back.
• Must-never-fail cases: the inputs where a wrong answer causes real harm. Few in number, graded strictly.
• Out-of-scope requests: what the feature should decline or redirect.

Label each case with a category, so results can be broken down. An average over a set that’s 90% easy cases says little about the 10% that drive complaints.

Sampling from real logs #

If your feature logs requests with a category and an outcome, a short script can build a balanced, reproducible sample. This one redacts obvious personal data, takes cases from every category, and over-samples requests where the user edited, retried or abandoned the output, because those are where the feature fell short.

import { readFile, writeFile } from 'node:fs/promises';

interface LoggedRequest {
  id: string;
  category: string;
  input: string;
  outcome: 'accepted' | 'edited' | 'retried' | 'abandoned';
}

// Redact before anything else. These patterns catch the obvious cases only: extend them for your data
// (names, addresses, account numbers) and have a person spot-check the output.
const redactions: [RegExp, string][] = [
  [/[\w.+-]+@[\w-]+\.[\w.-]+/g, '[email]'],
  [/\b(?:\d[ -]?){13,19}\b/g, '[card number]'],
  [/\+?\d[\d\s().-]{7,}\d/g, '[phone]'],
];
const redact = (text: string) => redactions.reduce((t, [pattern, label]) => t.replace(pattern, label), text);

/** Small seeded random generator, so the same logs always produce the same sample. */
function seededRandom(seed: number) {
  return () => {
    seed = (seed + 0x6d2b79f5) | 0;
    let t = Math.imul(seed ^ (seed >>> 15), 1 | seed);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}

const PER_CATEGORY = 20;
const HARD_SHARE = 0.5; // Half of each category from requests users weren't happy with.
const random = seededRandom(42);
const shuffle = <T,>(items: T[]) => items.map((item) => ({ item, key: random() })).sort((a, b) => a.key - b.key).map((x) => x.item);

const logs: LoggedRequest[] = (await readFile('logs/requests.jsonl', 'utf8'))
  .split('\n')
  .filter(Boolean)
  .map((line) => JSON.parse(line) as LoggedRequest);

const byCategory = Map.groupBy(logs, (r) => r.category);
const golden: (LoggedRequest & { reference: null })[] = [];

for (const [category, requests] of byCategory) {
  const hard = shuffle(requests.filter((r) => r.outcome !== 'accepted'));
  const easy = shuffle(requests.filter((r) => r.outcome === 'accepted'));
  const hardCount = Math.min(hard.length, Math.round(PER_CATEGORY * HARD_SHARE));
  const picked = [...hard.slice(0, hardCount), ...easy.slice(0, PER_CATEGORY - hardCount)];
  for (const r of picked) golden.push({ ...r, input: redact(r.input), reference: null });
  console.log(`${category}: ${picked.length} cases (${hardCount} hard) from ${requests.length} logged`);
}

// "reference" is filled in by a person in the next step: an ideal answer or a list of criteria.
await writeFile('evals/golden-draft.jsonl', `${golden.map((c) => JSON.stringify(c)).join('\n')}\n`);
console.log(`Wrote ${golden.length} cases to evals/golden-draft.jsonl`);

Two choices matter here. The seed makes the sample reproducible, so rebuilding the set from the same logs gives the same cases. And redaction happens first, before cases are written anywhere new. Regex redaction is a first pass, not a guarantee: always have a person review the draft for personal data before it’s shared or committed.

Reference answers and criteria #

Each case needs an expectation. There are two styles, and most sets use both:

• Reference answers: an ideal output written by an expert. Good when there’s essentially one right answer, such as an extracted value or a classification. Grading compares the output with the reference.
• Criteria: a short list of things a good output must and mustn’t do. Better for open-ended outputs like drafts and summaries, where many different outputs could be good. Grading checks each criterion.

Write criteria that someone else could apply without asking you:

• “Mentions that refunds take 5 to 10 business days” is checkable.
• “Is helpful and accurate” isn’t.

Have a second person review each expectation, especially for high-stakes cases. Disagreements about what “good” means are far cheaper to resolve here than after a launch. Rubrics and human grading covers how to make criteria consistent.

Privacy and freshness #

Privacy. Golden sets built from real use contain real people’s words. Treat them accordingly:

• Redact personal data, and have a person check the result.
• Store the set with the same access controls as production data.
• Respect deletion requests: if a user asks to delete their data, that includes eval cases built from it.
• Where possible, prefer paraphrasing sensitive cases over storing them verbatim.

See data privacy and retention for the wider picture.

Freshness. Users change, products change and the set slowly drifts from reality. Keep it current:

• Add every production failure as a case, once fixed, so it stays fixed.
• Refresh a sample regularly, for example monthly, from recent logs.
• Retire cases that no longer apply, such as questions about a removed feature, with a note on why.
• Version the set and record changes, so you can tell whether a score moved because the feature changed or the set did.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #