Data privacy and retention

Why this matters #

People tell AI features things they’d hesitate to write anywhere else: health worries, money problems, conflicts at work, details about customers and colleagues. Every one of those words passes through your system, usually reaches a model provider, and may land in logs, traces, eval sets and analytics.

Handling that data carelessly can break users’ trust, your contracts and the law. Handling it well is mostly a matter of knowing where data goes, sending and keeping as little as possible, and being honest with users about it. This article walks through each of those, with a practical technique for keeping personal details out of model requests. It isn’t legal advice: for regulated data or specific obligations, involve your legal and security teams early.

What data leaves your system #

Start by mapping every place user data travels in an AI feature. It’s usually more places than people expect:

For each destination, write down what data it receives, who can access it, how long it’s kept and how it’s deleted. That table is the core of any privacy review, and it frequently reveals the biggest risks are in your own logs and eval sets, not at the provider.

The provider’s side #

Model providers process your requests under their terms. Before sending user data, read:

• Whether inputs and outputs are used for training. Check the provider’s commercial terms for your plan.
• How long requests are retained, and whether you can shorten that. Anthropic documents its policies in API and data retention (opens in a new tab), including arrangements for organizations that need minimal retention; some features and models have their own retention requirements, so check before relying on one.
• Where data is processed, if you have data residency obligations.
• What agreements are available, such as data processing agreements, and for regulated data, whatever your industry requires.

Record the answers in your feature’s spec, and revisit them when you adopt a new provider, model or feature.

Retention and deletion #

Most data in AI systems is kept far longer than it’s useful. Set retention deliberately for each store:

• Traces with content: days or weeks, long enough to debug, not forever. Keep metadata (tokens, latency, stop reasons) longer if you need trends.
• Eval sets: only redacted or synthetic versions long term; review them for personal data before committing them to a repository.
• Conversation history and memory: as long as the user wants it, with clear controls to view and delete.
• Analytics: behavior only, pseudonymous IDs, no content at all.

And make deletion real: when a user deletes their account or a conversation, it should disappear from history, memory, traces with content, and any eval cases built from it. Map those paths before launch; retrofitting deletion across five systems is painful.

Sensitive and regulated data #

Some data needs extra care: health information, financial details, government ID numbers, children’s data, precise location, and anything covered by laws such as GDPR in the EU or sector rules like HIPAA for health data in the US. For these:

• Minimize. Ask whether the feature needs the sensitive field at all. A summary of a support ticket rarely needs the customer’s card number.
• Pseudonymize before sending. Replace identifiers with placeholders, send the request, and restore them in the output (below).
• Restrict access to traces and eval sets that could contain sensitive data.
• Get advice early. If the feature handles regulated data, involve legal and security before building, not after.

Pseudonymizing before sending #

A practical technique for many features: replace personal identifiers with placeholders before the request, and swap them back in your own code afterwards. The model still sees the structure (“[EMAIL_1] asked about order A1042”), but never the real values.

/**
 * Replaces emails, phone numbers and known names with placeholders before text is sent to a model,
 * and restores them in the model's output. The mapping never leaves your server.
 */
export function pseudonymize(text: string, knownNames: string[] = []) {
  const mapping = new Map<string, string>();
  const counters = new Map<string, number>();
  const placeholderFor = (kind: string, value: string) => {
    for (const [placeholder, original] of mapping) if (original === value) return placeholder;
    const n = (counters.get(kind) ?? 0) + 1;
    counters.set(kind, n);
    const placeholder = `[${kind}_${n}]`;
    mapping.set(placeholder, value);
    return placeholder;
  };

  let result = text.replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, (m) => placeholderFor('EMAIL', m));
  result = result.replace(/\+?\d[\d\s().-]{7,}\d/g, (m) => placeholderFor('PHONE', m));
  for (const name of knownNames) {
    result = result.replaceAll(name, () => placeholderFor('PERSON', name));
  }

  const restore = (output: string) => output.replace(/\[(EMAIL|PHONE|PERSON)_\d+\]/g, (p) => mapping.get(p) ?? p);
  return { text: result, restore };
}

const { text, restore } = pseudonymize('Hi, this is Dana Reyes (dana.reyes@example.com, +1 555 010 2030). Order A1042 is late.', ['Dana Reyes']);
console.log(text);
// Hi, this is [PERSON_1] ([EMAIL_1], [PHONE_1]). Order A1042 is late.

// ...send `text` to the model; suppose it replies:
const modelOutput = 'Hello [PERSON_1], sorry about order A1042. We will email [EMAIL_1] with tracking today.';
console.log(restore(modelOutput));
// Hello Dana Reyes, sorry about order A1042. We will email dana.reyes@example.com with tracking today.

Limits to keep in mind:

• Pattern matching misses things. Names you don’t know, addresses, IDs in unusual formats. Combine patterns with fields you already know (the customer’s name from your database) and review samples.
• Some features need the real data, for example a feature that checks whether an address is valid. Pseudonymize what the task doesn’t need.
• Placeholders change meaning slightly. Test that quality holds with pseudonymized inputs, using your eval set.

Telling users #

Users should know, in plain language and at the point of use, what happens to what they type:

• What’s sent where: “Your message is processed by our AI provider to generate a draft.”
• What’s kept, and for how long: “Drafts are stored for 30 days so you can find them again.”
• What it’s used for: whether anything is used to improve the product, and how to opt out.
• How to see and delete it: history, memory and an account-level delete that actually works.

Put a short version next to the feature and link to the full policy. Users trust features that are upfront more than ones that bury the details, and the act of writing the plain-language version often reveals practices you’d rather change.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #