Choosing a model: a decision framework

Why this matters #

Model choice affects nearly everything users notice: quality, speed and, through cost, whether the feature can exist at all. It’s also a decision you’ll revisit, because new models arrive regularly and old ones are retired. A repeatable way to choose saves you from re-arguing the decision every time, and from picking a model because of a benchmark headline.

This article gives you that process: start capable, measure on your own tasks, then trade down only where the evidence says you can.

The current lineup #

These are the current Claude models, generated from the wiki’s model configuration:

Within each model there’s another dial: effort, which controls how much the model reasons before answering. Lower effort is faster and cheaper; higher effort helps on hard problems. That means “which model?” is really “which model, at which effort?”

Start with the most capable model #

When you’re proving a feature works, use a capable model at a sensible effort level. The default in this wiki’s examples is Claude Opus 5.5. There are good reasons to start there:

• It answers the feasibility question honestly. If the feature doesn’t work with a capable model, a cheaper one won’t save it. If it does work, you know the ceiling.
• It makes your eval set meaningful. Outputs from a strong model help you write reference answers and spot which cases are genuinely hard.
• Optimizing is easier than rescuing. Trading down from a working feature is a measured, low-risk step. Debugging a feature that might be failing because of the model is not.

The most capable tier, Claude Fable 5.1, is worth testing for the hardest reasoning and long-running agent work, where its cost can be justified.

Compare on your own evals #

Public benchmarks tell you how models do on general tasks. Your feature succeeds or fails on your inputs and your quality bar. The reliable way to choose is to run candidate settings on your eval set and compare:

This script runs the same eval cases through several settings and prints pass rate, median latency and cost per success for each. The task here, extracting a date from a message, is deliberately simple so the code stays short; swap in your own cases and checks.

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic();

interface Setting {
  name: string;
  model: string;
  effort: 'low' | 'medium' | 'high';
  /** USD per million tokens, from the pricing page. */
  inputPerMTok: number;
  outputPerMTok: number;
}

const settings: Setting[] = [
  { name: 'Opus, medium', model: 'claude-opus-5-5', effort: 'medium', inputPerMTok: 4, outputPerMTok: 20 },
  { name: 'Opus, low', model: 'claude-opus-5-5', effort: 'low', inputPerMTok: 4, outputPerMTok: 20 },
  { name: 'Sonnet, low', model: 'claude-sonnet-5-5', effort: 'low', inputPerMTok: 2, outputPerMTok: 10 },
];

interface Case {
  input: string;
  expected: string;
}

// Real messages from your product, with the answer a careful human gave. Use dozens, not three.
const cases: Case[] = [
  { input: 'Can we move the review to next Tuesday? Today is Thursday 2026-10-15.', expected: '2026-10-20' },
  { input: 'Deadline is the last day of this month (it is 2026-02-03 today).', expected: '2026-02-28' },
  { input: 'Let us meet on the 3rd of March 2027.', expected: '2027-03-03' },
];

const SYSTEM = 'Extract the single date the message refers to. Reply with the date only, as YYYY-MM-DD.';

interface Result {
  passed: boolean;
  ms: number;
  costUsd: number;
}

async function runCase(s: Setting, c: Case): Promise<Result> {
  const start = performance.now();
  const response = await client.messages.create({
    model: s.model,
    max_tokens: 2048,
    output_config: { effort: s.effort },
    system: SYSTEM,
    messages: [{ role: 'user', content: c.input }],
  });
  const ms = performance.now() - start;
  const { input_tokens, output_tokens } = response.usage;
  const costUsd = (input_tokens * s.inputPerMTok + output_tokens * s.outputPerMTok) / 1_000_000;

  // Only a complete answer can pass; truncated or declined responses count as failures but still cost money.
  const text = response.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('').trim();
  const passed = response.stop_reason === 'end_turn' && text === c.expected;
  return { passed, ms, costUsd };
}

const median = (xs: number[]) => [...xs].sort((a, b) => a - b)[Math.floor(xs.length / 2)] ?? 0;

for (const s of settings) {
  const results: Result[] = [];
  for (const c of cases) results.push(await runCase(s, c));

  const passes = results.filter((r) => r.passed).length;
  const totalCost = results.reduce((sum, r) => sum + r.costUsd, 0);
  const perSuccess = passes === 0 ? 'n/a' : `$${(totalCost / passes).toFixed(5)}`;
  console.log(
    `${s.name.padEnd(14)} pass ${passes}/${cases.length}  median ${median(results.map((r) => r.ms)).toFixed(0)} ms  cost/success ${perSuccess}`,
  );
}

Run it with npx tsx evals/compare-settings.ts. When reading the results:

• Quality first. Discard any setting that falls below your quality bar, however cheap or fast.
• Then cost per success, then latency. Among the settings that pass, prefer the cheapest per success that fits your latency budget.
• Use enough cases. Three cases can’t distinguish settings. With a few dozen real cases, differences start to mean something; see golden datasets.

Effort and other dials #

Before switching to a smaller model, try lowering the effort on the model you have. It keeps one model, one set of prompts and one behavior to reason about, and on many tasks lower effort holds quality while cutting latency and cost. Anthropic’s guidance is to measure the most capable model at lower effort before building anything more complex, since lower effort on newer models often matches older models at higher effort (see the effort documentation (opens in a new tab)).

Effort guidance in short:

• Low: classification, extraction, short rewrites, chat. Fast and cheap.
• Medium: most product features. A sensible starting point.
• High and above: hard reasoning, complex code, long agent tasks, where correctness matters more than cost.

Effort is set per request with output_config: { effort }, so you can use different levels for different steps of the same feature.

Routing between models #

Some features have steps of very different difficulty: a quick classification, then a hard answer. Using a small model for the simple steps and a larger one for the hard steps can cut cost substantially. It also adds complexity:

• More to evaluate. Each step needs its own eval, and errors in an early step (such as routing to the wrong path) cascade.
• More to maintain. Two models means two sets of prompt quirks and two upgrade paths.
• Less caching. Caches are per model, so splitting traffic across models reduces cache reuse.

Route between models when your evals show a clear cost win at equal quality, not as a default architecture. The ticket router in When not to use AI is a good example of a step where a small model fits.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #