Latency budgets and perceived speed

Why this matters #

Users judge an AI feature by how fast it feels as much as by how good it is. A great answer that arrives after a long, silent wait often loses to a decent answer that starts appearing immediately. And unlike most of your stack, model calls can take seconds or more, so latency is a design constraint from day one.

This article explains where the time goes in a model call, how to set a latency budget for each kind of interaction, how to measure it in code, and how to make the same latency feel shorter.

Where the time goes #

A model call’s latency has several parts, and they respond to different fixes:

Two numbers summarize it:

• Time to first token (TTFT): how long until the first visible output appears. This is what users feel as “is it working?”
• Total time: how long until the response is complete. This matters when nothing useful can happen until the end, such as structured output your code parses.

Each part has its own lever. Output length is often the biggest and most controllable: a response twice as long takes roughly twice as long to write. Thinking time grows with the effort setting and the difficulty of the task. Large, repeated inputs can be cached. Model choice affects all of them.

Setting a latency budget #

A latency budget is the time an interaction can take before the experience suffers. It depends on what the user is doing, not on what the model can manage.

A useful anchor is the classic research on response times summarized by Nielsen Norman Group: about 0.1 seconds feels instant, about 1 second keeps the user’s flow of thought, and about 10 seconds is the limit of attention before people want to do something else (Response times: the 3 important limits (opens in a new tab)). Applied to AI features, these rules of thumb work well:

The budgets are opinions grounded in those limits, not standards. What matters is writing one down per feature, so “is this fast enough?” has an answer. Inline features usually can’t afford a large model call per keystroke at all; they need a small, fast model, precomputed results, or a different design.

Measuring time to first token #

Measure latency with the model, prompt and input sizes you’ll use in production. This script streams a response, records when the first visible text arrives, and reports the median over a few runs.

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

interface Timing {
  /** Milliseconds until the first visible text. Null if no text arrived. */
  firstTextMs: number | null;
  totalMs: number;
  outputTokens: number;
  stopReason: Anthropic.Message['stop_reason'];
}

async function timeRequest(prompt: string): Promise<Timing> {
  const start = performance.now();
  let firstTextMs: number | null = null;

  const stream = client.messages.stream({
    model: 'claude-opus-5-5',
    // Thinking is always on for this model and counts towards max_tokens, so leave headroom.
    max_tokens: 4096,
    output_config: { effort: 'low' },
    messages: [{ role: 'user', content: prompt }],
  });

  for await (const event of stream) {
    // Thinking happens before the first text, so this measures what users actually wait for.
    if (firstTextMs === null && event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
      firstTextMs = performance.now() - start;
    }
  }

  const message = await stream.finalMessage();
  return {
    firstTextMs,
    totalMs: performance.now() - start,
    outputTokens: message.usage.output_tokens,
    stopReason: message.stop_reason,
  };
}

const median = (values: number[]) => {
  const sorted = [...values].sort((a, b) => a - b);
  return sorted[Math.floor(sorted.length / 2)] ?? 0;
};

const prompt = 'Summarize in three bullet points why latency matters for AI features.';
const runs: Timing[] = [];
// Sequential on purpose: parallel requests would measure your concurrency, not a single user's wait.
for (let i = 0; i < 5; i++) runs.push(await timeRequest(prompt));

// A run that didn't end normally (truncated, refused, paused) isn't a valid latency sample.
const valid = runs.filter((r) => r.stopReason === 'end_turn' && r.firstTextMs !== null);
if (valid.length < runs.length) {
  console.warn(`Ignored ${runs.length - valid.length} run(s) that didn't finish normally:`, runs.map((r) => r.stopReason));
}

console.log(`Median time to first text: ${median(valid.map((r) => r.firstTextMs ?? 0)).toFixed(0)} ms`);
console.log(`Median total time: ${median(valid.map((r) => r.totalMs)).toFixed(0)} ms`);
console.log(`Median output tokens: ${median(valid.map((r) => r.outputTokens))}`);

Run it with npx tsx scripts/measure-latency.ts. A few things to keep in mind when reading the numbers:

• Five runs give a rough picture, not a guarantee. Latency varies with load. Before launch, look at many requests and watch the slow tail (the 90th or 95th percentile), not only the median. Tracing records this in production.
• Measure from where users are. Your server’s latency to the API is only part of it; add your own backend and the user’s network.
• Test real input sizes. A prompt with a long document takes longer to read than a one-line test prompt.

Making it feel faster #

Once you’ve measured, there are two kinds of improvement: make it faster, and make it feel faster.

Make it faster:

• Ask for shorter output. It’s often the biggest lever. “In three bullet points” is faster than an open-ended answer.
• Lower the effort for tasks that don’t need deep reasoning. Less thinking means an earlier first token. Measure the quality impact on your evals.
• Use a smaller model for simple, latency-sensitive steps such as classification or routing. See choosing a model.
• Cache repeated input. With prompt caching (opens in a new tab), a long system prompt or document you send on every request is processed faster and costs less after the first time.
• Do work ahead of time. Precompute summaries or tags in the background so the interactive path only reads results.

Make it feel faster:

• Stream the response so text appears as it’s generated. For conversational features, this turns a long wait into a short one.
• Show progress that means something: “Reading 3 documents… Drafting reply…” beats a spinner.
• Show partial value early: results as they arrive, a skeleton of the final layout, or the sources before the answer.
• Keep the rest of the interface usable while the AI works.

Streaming and loading states covers the UX side in detail.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #