LLM-as-judge: design and calibration

Why this matters #

Human grading is the reference standard for qualities like correctness, tone and faithfulness to sources, but it doesn’t scale to every prompt change, every model comparison and every night’s regression run. A model can apply the same rubric in minutes, for a fraction of the cost.

The catch is that a model judge is itself an AI system: it can be wrong, inconsistent and biased in predictable ways. Used carefully, with one criterion at a time, structured verdicts and regular calibration against human grades, it’s one of the most useful tools in the eval stack. Used carelessly, it gives you confident numbers that don’t mean anything. This article shows the careful version.

When to use a model as judge #

A model judge sits in the middle of the eval pyramid: above code checks, below human grading. Use it when:

• Code can’t check the quality. Faithfulness to sources, whether a question was actually answered, tone, completeness.
• Humans can’t grade the volume. Hundreds of outputs per change, or nightly runs.
• You have human grades to calibrate against. Without them, you can’t know whether the judge is right.

Don’t use a judge for things code can check exactly, such as format, length, required phrases or valid JSON. Code is faster, free and never wrong about those.

Writing a judge prompt #

Judges work best with a narrow, well-defined job. The principles:

• One criterion per judgment. “Is it grounded?” and “is the tone right?” are separate calls. A judge asked to score five things at once does all of them worse.
• Use your human rubric. The same definition, pass example, fail example and borderline rule that your human graders use. If humans can’t agree on a criterion, a model won’t either.
• Ask for a pass/fail verdict, not a score. Binary verdicts are more consistent than 1 to 10 scales.
• Ask for a short justification first. Having the judge quote the part of the output that decides the verdict, before giving it, tends to make verdicts more careful and makes disagreements easier to diagnose.
• Give the judge what it needs and nothing else. For groundedness, the sources and the answer. Not the prompt version, not which model produced it.
• Return structured output, so verdicts are parsed reliably.

A judge in TypeScript #

This judge grades one criterion, groundedness, and returns a structured verdict. A calibration function then compares its verdicts with human labels on the same outputs.

import Anthropic from '@anthropic-ai/sdk';
import { zodOutputFormat } from '@anthropic-ai/sdk/helpers/zod';
import { z } from 'zod';

const client = new Anthropic();

// The reasoning field comes first, so the judge explains before it decides.
const VerdictSchema = z.object({
  reasoning: z.string().describe('One or two sentences quoting the part of the answer that decides the verdict'),
  verdict: z.enum(['pass', 'fail']),
});

type Verdict = z.infer<typeof VerdictSchema>['verdict'];

// The same definition, examples and borderline rule human graders use.
const GROUNDED = [
  'Criterion: grounded. Every factual claim in the answer is supported by the sources.',
  'Pass: all claims appear in the sources, possibly reworded.',
  'Fail: any claim (a date, amount, policy or step) that the sources do not support.',
  'Borderline: courtesy phrases such as apologies are not factual claims; ignore them.',
].join('\n');

/** Grades one answer on one criterion. Returns null when the judge didn't finish, so it's never counted as a pass. */
export async function judgeGrounded(sources: string, answer: string): Promise<Verdict | null> {
  const response = await client.messages.parse({
    model: 'claude-sonnet-5-5',
    max_tokens: 2048,
    // No fallback on purpose: switching judges mid-run would make scores incomparable.
    output_config: { effort: 'low', format: zodOutputFormat(VerdictSchema) },
    system: `You grade one criterion for an evaluation. Apply it strictly.\n\n${GROUNDED}`,
    messages: [{ role: 'user', content: `<sources>\n${sources}\n</sources>\n\n<answer>\n${answer}\n</answer>` }],
  });
  if (response.stop_reason !== 'end_turn' || !response.parsed_output) return null;
  return response.parsed_output.verdict;
}

interface LabeledExample {
  sources: string;
  answer: string;
  /** The verdict agreed by human graders. */
  human: Verdict;
}

/** Compares judge verdicts with human labels. False passes are the dangerous disagreements. */
export async function calibrate(examples: LabeledExample[]) {
  let agree = 0;
  let falsePass = 0;
  let falseFail = 0;
  let unfinished = 0;
  for (const example of examples) {
    const verdict = await judgeGrounded(example.sources, example.answer);
    if (verdict === null) unfinished++;
    else if (verdict === example.human) agree++;
    else if (verdict === 'pass') falsePass++;
    else falseFail++;
  }
  const graded = examples.length - unfinished;
  return {
    agreement: graded === 0 ? 0 : agree / graded,
    falsePass,
    falseFail,
    unfinished,
  };
}

Two design choices to notice:

• Unfinished judgments aren’t passes. If the judge is cut off or declines, the result is null, and calibration reports it separately. A silent default to “pass” would inflate every score.
• Disagreements are split by direction. A judge that passes outputs humans failed (a false pass) hides real problems; one that fails outputs humans passed (a false fail) wastes your time. You usually care much more about the first.

This example uses Claude Sonnet 5.5 as the judge, a common choice for grading at volume because it’s cheaper than the model being evaluated. Using the same model works too. Either way, the judge earns trust only through calibration.

Calibration and bias #

Calibration means checking the judge against human grades on the same outputs, and improving it until they agree well enough:

Set a target before calibrating: for example, at least 90% agreement with humans and no false passes on must-never-fail criteria. Until the judge meets it, treat its scores as rough signals, not decisions.

Model judges also have known biases. Research on LLM judges has documented several (see Zheng et al., 2023 (opens in a new tab)):

• Position bias: when comparing two outputs side by side, judges can favor whichever comes first (or second). Mitigation: run comparisons in both orders, or grade each output on its own against the rubric.
• Length bias: longer, more detailed-looking answers can be favored even when they’re not better. Mitigation: criteria that don’t reward length, and a separate length check in code.
• Self-preference: a judge may favor outputs that resemble its own style. Mitigation: calibrate against human grades, which reveals the bias if it affects your criteria.

The common thread: grade against a rubric, one criterion at a time, and verify against humans. Biases mostly show up in open-ended comparisons (“which is better?”), and much less in narrow, well-defined pass/fail criteria.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #