Product analytics events for AI features

Why this matters #

On launch day, someone will ask “is it working?” Without the right events, you’ll answer with page views and button clicks, which tell you people are curious, not whether they’re helped. The metrics that matter for AI features, such as acceptance, edits, retries and cost per success, need a few events most analytics plans forget.

Instrumenting them takes an afternoon if you do it before launch, and is painful to retrofit after. This article lists the events every AI feature needs, gives you a typed schema that includes model, prompt version and cost, shows how to keep user content out of analytics, and maps the events to the questions they answer.

The events every AI feature needs #

Think of each AI output as moving through a short journey. Each step is an event:

“Abandoned” is usually inferred rather than tracked: a completed output with no outcome event within a session.

An event schema #

Every event shares a base: which feature, which configuration (prompt version, model, A/B variant) and a request ID linking events for the same output together. Validating events at the point they’re sent catches instrumentation bugs early, before they pollute months of data.

import { z } from 'zod';

// Shared by every event: what produced the output, and an ID tying events for one output together.
const Base = z.object({
  feature: z.string(),
  requestId: z.string(),
  promptVersion: z.string(),
  model: z.string(),
  variant: z.string().optional(),
  userId: z.string(), // Pseudonymous ID, never an email or name.
});

const AiEvent = z.discriminatedUnion('type', [
  Base.extend({ type: z.literal('ai_requested') }),
  Base.extend({
    type: z.literal('ai_completed'),
    outcome: z.enum(['end_turn', 'max_tokens', 'refusal', 'other_stop', 'error', 'cancelled']),
    latencyMs: z.number().nonnegative(),
    inputTokens: z.number().int().nonnegative(),
    outputTokens: z.number().int().nonnegative(),
    costUsd: z.number().nonnegative(),
  }),
  Base.extend({ type: z.literal('ai_accepted'), editRatio: z.number().min(0).max(1) }),
  Base.extend({ type: z.literal('ai_retried'), reason: z.string().max(100).optional() }),
  Base.extend({ type: z.literal('ai_dismissed'), reason: z.string().max(100).optional() }),
  Base.extend({ type: z.literal('ai_feedback'), value: z.enum(['up', 'down']), reason: z.string().max(100).optional() }),
]);

export type AiEvent = z.infer<typeof AiEvent>;

/**
 * Validates and sends an event. Content (prompts, outputs, documents) is deliberately not part of the schema:
 * analytics records behavior, not what users wrote. Free-text reasons are short, chosen from a list in the UI.
 */
export function track(event: AiEvent): void {
  const parsed = AiEvent.safeParse(event);
  if (!parsed.success) {
    // An invalid event is an instrumentation bug: surface it in development, never send bad data.
    console.error('Invalid analytics event', parsed.error.issues);
    return;
  }
  // Replace with your analytics client, e.g. analytics.capture(parsed.data.type, parsed.data).
  console.log(JSON.stringify(parsed.data));
}

const base = { feature: 'reply-drafts', requestId: 'req_123', promptVersion: 'reply-v4', model: 'claude-opus-5-5', userId: 'u_8f2c' };
track({ ...base, type: 'ai_requested' });
track({ ...base, type: 'ai_completed', outcome: 'end_turn', latencyMs: 3120, inputTokens: 2900, outputTokens: 410, costUsd: 0.0198 });
track({ ...base, type: 'ai_accepted', editRatio: 0.12 });

Three design choices are deliberate:

• The base fields make every event attributable. When a prompt change rolls out, you can split every metric by promptVersion; when you A/B test, by variant.
• Outcome is an enum, not free text. It maps directly to the states your UI designs for, so dashboards can show truncation and refusal rates without parsing strings.
• There’s no field for content. You can’t accidentally send a user’s email to your analytics tool if the schema has nowhere to put it.

Privacy by design #

Analytics tools are often shared widely inside a company and kept for years. Treat them accordingly:

• Log behavior, not content. Events say that a draft was accepted with 12% edits, not what it said.
• Use pseudonymous IDs. An internal user ID, never an email address or name.
• Keep free text short and structured. Offer reasons from a list (“Wrong facts”, “Wrong tone”, “Too long”) instead of open text boxes that invite personal details.
• Store content separately, if at all. If you need examples for debugging or evals, keep them in your tracing system with stricter access and shorter retention, and with consent where required. See tracing and data privacy.

Questions the events answer #

With these events in place, the metrics from defining success metrics become straightforward queries:

Exact definitions belong in one place, such as the metric definitions reference, so every dashboard computes them the same way.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #