Eval types cheat sheet
Every kind of eval in one place, from code checks and rubrics to human grading, LLM-as-judge and online metrics, with when to use each.
2 min read Updated
On this page
The eval types compared
| Type | Checks | Cost and speed | Reliability | Use for | Full article |
|---|---|---|---|---|---|
| Code checks | Format, length, required facts, valid JSON, forbidden phrases | Free, instant | Exact for what it checks | Everything that can be checked exactly; every change | Why evals are product metrics |
| Reference comparison | Output matches a known answer | Free, instant | High when there’s one right answer | Extraction, classification, dates, IDs | Golden datasets |
| Human grading | Any criterion an expert can judge | Slow, expensive | The reference standard, if graders agree | Specialist correctness, must-never-fail, calibrating judges | Rubrics and human grading |
| LLM-as-judge | Criteria code can’t check: grounding, tone, completeness | Cheap, fast at scale | Only as good as its calibration | Nightly suites, large comparisons | LLM-as-judge |
| Pairwise comparison | Which of two outputs is better | Medium | Prone to position and length bias | Comparing variants when absolute criteria are hard | LLM-as-judge |
| Online metrics | What real users do: accept, edit, retry | Slow, needs traffic | The final word on value, but noisy | Confirming offline results after launch | Capturing feedback |
| A/B test | Which variant performs better with real users | Slow, needs volume | Strongest evidence of impact | Decisions with quality and cost trade-offs | A/B testing AI features |
Choosing a grader for each criterion
- Can code check it exactly? Use a code check.
- Is there one right answer? Compare with a reference answer.
- Does it need judgment, at volume? Use a calibrated LLM judge.
- Does it need expertise, or is it must-never-fail? Use human grading, and use those grades to calibrate the judge.
- Does it depend on real user behavior? Measure it online, and confirm big decisions with an A/B test.
Where each runs
| When | What runs |
|---|---|
| Every prompt or model pull request | Code checks and reference comparisons on a smoke set, plus all must-never-fail cases |
| Nightly | The full suite, including LLM-judge criteria |
| Each significant change | A small human grading session; re-calibrate the judge if needed |
| After launch, continuously | Online metrics, split by prompt version |
| Big decisions | An A/B test with a pre-registered decision rule |
Numbers to watch
- Pass rate per category, not just overall.
- Must-never-fail pass rate: 100%, always.
- Judge agreement with humans, and false passes in particular.
- Grader agreement (Cohen’s kappa) for human-graded criteria.
- Held-out pass rate, to catch overfitting to the cases you tuned against.
Definitions are on the metric definitions page.