Context windows, retrieval and memory trade-offs

Why this matters #

A model can only work with what it sees. If the right information isn’t in the request, the model will either say it doesn’t know or, worse, produce something plausible and wrong. Many “the AI is dumb” complaints are really context problems: the model was never shown what it needed.

Deciding what goes into the context is one of the most important design choices in an AI feature. It drives quality, cost and latency at the same time. This article explains the options, when each fits, and how caching changes the economics.

What a context window is #

The context window is the maximum amount of text, measured in tokens, a model can take into account in one request: the system prompt, tool definitions, conversation history, documents and the response itself. Current Claude models have context windows of 1M tokens, which is roughly the length of several long books.

A large window is useful, but three things make “just send everything” a poor default:

• Cost: you pay for every input token on every request. Sending a large document with each question multiplies cost by the number of questions.
• Latency: the model has to read the input before it can answer. More input means a later first token.
• Focus: irrelevant material can distract from what matters. Models are good at finding the relevant part of a long input, but a tightly chosen context still tends to give more reliable answers, and it makes answers easier to cite and check.

So the question isn’t “does it fit?” but “what’s the smallest context that reliably contains what’s needed?”

Long context or retrieval? #

There are two broad strategies for getting knowledge into a request:

• Long context: send the whole document or dataset with the request. Simple, nothing is missed, and the model can connect information across the whole thing.
• Retrieval (often called RAG, retrieval-augmented generation): search your content for the parts relevant to this request and send only those. Scales to far more content than any window, keeps requests small, and makes it natural to cite sources. But if retrieval misses the right passage, the model can’t use it.

In practice many features combine both: retrieval narrows a large corpus to a few documents, and those documents go into the context whole. Whatever you choose, measure it: build a set of real questions with known answers and check how often the right information reaches the model. That retrieval eval is part of Project 2.

Memory across sessions #

Model APIs are stateless: each request contains everything the model knows about the conversation. Anything that feels like memory is something your application stores and sends back. That gives you three design decisions:

1. What to remember. Preferences (“always reply in British English”), facts the user stated, decisions made. Not everything: storing every message forever costs money, slows requests and raises privacy questions.
2. How to bring it back. A short list of saved facts in the system prompt, retrieval over past conversations, or a running summary that replaces old messages.
3. What users control. People should be able to see what’s remembered, correct it and delete it. Memory that users can’t inspect feels unsettling when it surfaces something unexpected. See Data privacy and retention.

For long conversations within one session, the API offers server-side compaction (opens in a new tab), which summarizes earlier turns automatically as the conversation approaches its limits.

Cost and latency of context #

Because you pay for input on every request, context size dominates cost in context-heavy features. Prompt caching changes that. When many requests share the same beginning, such as a long system prompt or a document users ask several questions about, the API can cache that prefix. Later requests that start with the same prefix read it from the cache at a reduced price and with less processing time.

This example answers several questions about one long document. The document sits in the system prompt with a cache marker, so only the first request pays the full price to process it.

import Anthropic from '@anthropic-ai/sdk';
import { readFile } from 'node:fs/promises';

const client = new Anthropic();
const document = await readFile('handbook.md', 'utf8');

async function ask(question: string) {
  const response = await client.messages.create({
    model: 'claude-opus-5-5',
    max_tokens: 4096,
    output_config: { effort: 'low' },
    system: [
      { type: 'text', text: 'Answer questions using only the handbook below. If it does not say, reply "Not covered".' },
      // Everything up to and including this block is cached. Keep it identical across requests:
      // any change, even a timestamp, starts a new cache entry.
      { type: 'text', text: document, cache_control: { type: 'ephemeral' } },
    ],
    messages: [{ role: 'user', content: question }],
  });

  const answer = response.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('');
  const { cache_creation_input_tokens: written, cache_read_input_tokens: read, input_tokens: uncached } = response.usage;
  console.log(`\nQ: ${question}`);
  console.log(response.stop_reason === 'end_turn' ? `A: ${answer}` : `(no complete answer: ${response.stop_reason})`);
  console.log(`tokens → cache write: ${written ?? 0}, cache read: ${read ?? 0}, uncached: ${uncached}`);
}

// The first call writes the cache; the following calls should show large "cache read" numbers.
for (const q of ['How many vacation days do new hires get?', 'What is the expense limit for travel?', 'Who approves remote work?']) {
  await ask(q);
}

Run it with any long Markdown file as handbook.md. On the second and third questions you should see most of the document counted as cache reads. If cache reads stay at zero, something in the cached prefix is changing between requests, or the prefix is shorter than the model’s minimum cacheable length. The caching documentation (opens in a new tab) lists both.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #