Tracing and observability for LLM calls

Why this matters #

When a user reports that “the AI gave me a weird answer”, you need to see exactly what happened: what went in, what came out, which prompt and model produced it, how it ended, how long it took and what it cost. Without that, the report becomes a shrug and a guess. With it, it becomes a bug fix and a new eval case.

Traces are also how you watch the feature as a system: latency creeping up, refusals spiking after a prompt change, cost per request doubling because someone added a long document to the context. This article covers what a trace should contain, a small wrapper that records one per call, a habit for reviewing them, and the alerts worth setting.

What a trace should contain #

A trace is the record of one AI request, from your code’s point of view. For each model call, record:

Two of these deserve emphasis:

• The prompt version. Without it, you can’t tell whether a spike in bad outputs started with a prompt change.
• The provider’s request ID. Every API response carries one. If you ever need to ask the provider about a specific request, it’s the first thing they’ll want.

Content is the sensitive part. Inputs and outputs are what make traces useful for debugging, and they’re user data. Decide deliberately: store content for a sample or for internal users only, keep it in a system with restricted access and short retention, redact what you can, and follow your privacy rules. Many teams keep metadata for every request and content for a small, consented sample.

Instrumenting calls #

A thin wrapper around the SDK call is enough to record a trace for every request. This one records context, outcome, tokens, latency and the request ID, and stores content only when you say so:

import Anthropic from '@anthropic-ai/sdk';
import { randomUUID } from 'node:crypto';

const client = new Anthropic();

export interface Trace {
  traceId: string;
  feature: string;
  promptVersion: string;
  model: string;
  startedAt: string;
  latencyMs: number;
  outcome: string;
  requestId?: string | null | undefined;
  inputTokens?: number;
  outputTokens?: number;
  cacheReadTokens?: number | null;
  /** Only when allowed by policy, and stored with restricted access and short retention. */
  content?: { input: Anthropic.MessageParam[]; output: string };
}

type Sink = (trace: Trace) => void;
// Replace with your tracing backend. A JSON line per request is already searchable in most log tools.
const logSink: Sink = (trace) => console.log(JSON.stringify(trace));

interface TraceOptions {
  feature: string;
  promptVersion: string;
  storeContent?: boolean;
  sink?: Sink;
}

export async function tracedCreate(
  params: Anthropic.MessageCreateParamsNonStreaming,
  { feature, promptVersion, storeContent = false, sink = logSink }: TraceOptions,
): Promise<Anthropic.Message> {
  const startedAt = new Date();
  const base = { traceId: randomUUID(), feature, promptVersion, model: params.model, startedAt: startedAt.toISOString() };
  try {
    const message = await client.messages.create(params);
    const output = message.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('');
    sink({
      ...base,
      latencyMs: Date.now() - startedAt.getTime(),
      outcome: String(message.stop_reason),
      requestId: message._request_id,
      inputTokens: message.usage.input_tokens,
      outputTokens: message.usage.output_tokens,
      cacheReadTokens: message.usage.cache_read_input_tokens,
      ...(storeContent && { content: { input: params.messages, output } }),
    });
    return message;
  } catch (error) {
    // Record failures too: error spikes are one of the most useful things a trace shows.
    sink({
      ...base,
      latencyMs: Date.now() - startedAt.getTime(),
      outcome: error instanceof Anthropic.APIError ? `error ${error.status ?? 'network'}` : 'error unknown',
      requestId: error instanceof Anthropic.APIError ? error.requestID : undefined,
    });
    throw error;
  }
}

// Usage: same parameters as client.messages.create, plus trace context.
const message = await tracedCreate(
  {
    model: 'claude-opus-5-5',
    max_tokens: 4096,
    output_config: { effort: 'low' },
    messages: [{ role: 'user', content: 'Summarize: the launch moved to the 19th because legal needs a week.' }],
  },
  { feature: 'thread-summary', promptVersion: 'summary-v2' },
);
// The caller still handles every stop reason, as in every other example on this site.
console.log(message.stop_reason);

Things to notice:

• Errors are traced too, with the HTTP status and request ID, then re-thrown so the caller handles them as usual.
• Content is opt-in. The default records metadata only.
• The wrapper doesn’t change behavior. It returns the same message the SDK would; stop-reason handling stays in your feature code.
• The same idea works for streaming: record the trace after finalMessage(), as the day-one demo route does in its log line.

If you already run an observability backend, you may not need to write this yourself. The Claude SDKs and API can emit OpenTelemetry (opens in a new tab) spans for Messages API requests, following the OpenTelemetry GenAI conventions, so they appear alongside your application’s own traces. It’s in beta at the time of writing, so check the docs for current setup. You’ll still want to attach your own context, such as feature and prompt version.

Reviewing traces #

Traces are most valuable when someone reads them regularly, not only during incidents. A weekly habit that takes 30 minutes:

1. Sample broadly: 20 random traces from the week, plus every trace with an error, a refusal, a truncation, or a thumbs-down, edit or retry linked to it through the request ID in your analytics events.
2. Read inputs and outputs, where you’re allowed to. Note anything surprising: requests outside the intended scope, outputs that are technically fine but unhelpful, unusually long inputs.
3. Tag and count, as in capturing feedback.
4. Act: new eval cases for real failures, bug tickets for integration problems, and questions for the next product review.

Reading real traces is also the fastest way to keep your intuition calibrated. Teams that never read traces gradually lose touch with what users actually ask.

Monitoring and alerts #

From the same traces, build a small dashboard and a few alerts:

Break every chart down by prompt version and model. Most regressions line up with a change, and seeing that immediately saves hours. Cost views get their own article: cost dashboards and budgets.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #