Rubrics and human grading

Why this matters #

Many of the qualities users care most about can’t be checked with code: is the answer correct, is it complete enough to act on, is the tone right, does it stay faithful to the sources? Someone has to judge them. Human grading is how you do that, and it’s also the reference standard every automated grader is measured against.

But human judgment is only useful when it’s consistent. If two graders look at the same output and one says “pass” while the other says “fail”, the score measures the graders, not the feature. A good rubric fixes that, and a simple agreement check tells you whether it worked. This article covers both.

What makes a rubric gradeable #

A rubric is a list of criteria, each with a clear definition of what passes. Good criteria share four properties:

• Specific: “states the refund window from the policy” rather than “is accurate”.
• Observable: a grader can point to the part of the output that passes or fails it.
• Independent: each criterion measures one thing. “Correct and concise” is two criteria.
• Anchored with examples: a passing example, a failing example, and guidance for borderline cases.

Start with three to six criteria per feature. More than that and graders get slow and inconsistent; fewer and you may miss what matters. Mark any criterion that must never fail, like “doesn’t promise anything outside the policy”, so it’s graded and reported separately.

Pass/fail or scales? #

Two common scoring formats:

Prefer pass/fail per criterion for most product evals. It’s faster to grade, graders agree more often, and the result, “92% of outputs are grounded”, is easy to act on. If you need nuance, split a criterion in two rather than adding a scale: “answers the question” and “answers it completely” instead of “helpfulness 1 to 5”.

Running a grading session #

A grading session can be an hour with two people and a spreadsheet:

1. Choose graders who know the domain. Support leads for support replies, clinicians for medical summaries. Engineers can grade format; they shouldn’t be the only judges of correctness in a specialist domain.
2. Calibrate first. Grade five outputs together, discuss every disagreement, and refine the rubric’s examples and borderline rules.
3. Grade independently. Each grader scores the same set without seeing the other’s scores.
4. Grade blind. Hide which prompt version or model produced each output, and shuffle the order, so graders can’t favor the version they expect to win.
5. Measure agreement on the overlap (next section).
6. Resolve disagreements by discussion, and update the rubric so the same disagreement doesn’t recur.

Keep the graded outputs. They become the reference labels for calibrating an LLM judge, and the baseline for later comparisons.

Measuring agreement #

Before trusting human grades, check that two graders would score the same output the same way. Two numbers are useful:

• Percent agreement: how often both graders gave the same verdict. Simple, but inflated when most outputs pass, because graders agree by chance.
• Cohen’s kappa: agreement corrected for chance. 1 means perfect agreement, 0 means no better than chance. A widely used rule of thumb (Landis and Koch, 1977) treats values above about 0.6 as substantial agreement and above 0.8 as almost perfect, though what’s acceptable depends on the stakes.

This function computes both for pass/fail grades on the same outputs:

type Verdict = 'pass' | 'fail';

/** Percent agreement and Cohen's kappa for two graders' verdicts on the same outputs, in the same order. */
export function agreement(a: Verdict[], b: Verdict[]) {
  if (a.length !== b.length || a.length === 0) throw new Error('Both graders must grade the same, non-empty set.');
  const n = a.length;
  const observed = a.filter((v, i) => v === b[i]).length / n;

  // Agreement expected by chance, from each grader's own pass rate.
  const passA = a.filter((v) => v === 'pass').length / n;
  const passB = b.filter((v) => v === 'pass').length / n;
  const expected = passA * passB + (1 - passA) * (1 - passB);

  const kappa = expected === 1 ? 1 : (observed - expected) / (1 - expected);
  return { percentAgreement: observed, kappa };
}

// Two graders, ten outputs, one criterion ("grounded").
const graderA: Verdict[] = ['pass', 'pass', 'fail', 'pass', 'pass', 'fail', 'pass', 'pass', 'pass', 'fail'];
const graderB: Verdict[] = ['pass', 'pass', 'fail', 'pass', 'fail', 'fail', 'pass', 'pass', 'pass', 'pass'];
const { percentAgreement, kappa } = agreement(graderA, graderB);
console.log(`Agreement ${(percentAgreement * 100).toFixed(0)}%, kappa ${kappa.toFixed(2)}`);
// Agreement 80%, kappa 0.52: the graders agree often, but less than it seems once chance is removed.

The example shows why kappa matters. 80% agreement sounds good, but because both graders pass most outputs, much of that agreement would happen by chance. A kappa of 0.52 says the criterion needs clearer examples or a better borderline rule before its grades can be trusted.

Compute agreement per criterion, not overall: one vague criterion can hide behind several clear ones.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #