Metric definitions
Exact definitions and formulas for the metrics used in this wiki, including task success, acceptance, edit rate, retry rate and cost per success.
On this page
How to use this page
One definition per metric, so every dashboard, report and discussion computes it the same way. Each entry gives the formula, the events it needs (from the analytics event schema), what it tells you, and how it misleads. Adapt the definitions to your product, then treat your version as the source of truth.
Behavior metrics
| Behavior metric | Formula | Events needed |
|---|---|---|
| Discovery rate | Users who requested output ÷ users who saw the entry point | ai_entry_shown, ai_requested |
| Completion rate | Requests that ended with end_turn ÷ requests |
ai_requested, ai_completed |
| Acceptance rate | Outputs accepted ÷ outputs completed | ai_completed, ai_accepted |
| Average edit ratio | Mean edit ratio over accepted outputs | ai_accepted (editRatio) |
| Heavy-edit rate | Accepted outputs with edit ratio above 0.5 ÷ accepted outputs | ai_accepted (editRatio) |
| Retry rate | Outputs regenerated ÷ outputs completed | ai_completed, ai_retried |
| Abandonment rate | Completed outputs with no outcome event in the session ÷ completed | ai_completed, outcome events |
| Second-use rate | Users who use the feature again within 7 days ÷ users with a first successful use | ai_accepted with user and time |
What they tell you: whether people find the feature, whether it works when they try it, whether the output is useful, and whether they come back.
How they mislead:
- Acceptance without edit ratio counts a fully rewritten draft as a success. Always report them together.
- Retry rate can rise for good reasons (users exploring alternatives). Read it alongside acceptance.
- Abandonment depends on how you define a session. Fix the definition and keep it stable.
- Discovery measures placement and labeling as much as value.
Quality metrics
| Quality metric | Formula | Source |
|---|---|---|
| Eval pass rate | Cases passing all criteria ÷ cases, per category | Eval suite |
| Must-never-fail pass rate | Must-never-fail cases passing ÷ must-never-fail cases (target 100%) | Eval suite |
| Held-out pass rate | Pass rate on cases never used for tuning | Held-out set |
| Judge agreement | Judge verdicts matching human labels ÷ graded examples | Calibration |
| Truncation rate | Completions with max_tokens or context window stop ÷ completions |
Traces |
| Refusal rate | Completions with refusal ÷ completions |
Traces |
How they mislead: an overall pass rate can hide a failing category; a pass rate on cases you tuned against overstates quality; a judge’s scores mean only as much as its calibration.
Cost metrics
| Cost metric | Formula | Data from |
|---|---|---|
| Cost per request | Total model cost ÷ requests | Traces (usage × prices) |
| Cost per success | Total model cost ÷ successful outcomes (for example, accepted outputs) | Traces and analytics |
| Cost per active user | Total model cost ÷ users with at least one request in the period | Traces and analytics |
| Margin per user | (Revenue − model cost − other costs to serve) ÷ users, per segment | Billing, traces |
| Waste share | Model cost of failed, retried and abandoned outputs ÷ total model cost | Traces and analytics |
How they mislead:
- Cost per request hides the cost of failures and retries; use cost per success for decisions.
- Averages across users hide heavy users who lose money; segment by usage. See unit economics.
- Prices change. Compute cost from token counts and the current price configuration, not from a hard-coded rate.
Performance metrics
| Performance metric | Formula | Data from |
|---|---|---|
| Time to first text | Time from request to first visible output, median and 95th percentile | Traces or client timing |
| Total latency | Time from request to final event, median and 95th percentile | Traces |
| Error rate | Requests ending in an API or network error ÷ requests | Traces |
Report the slow tail (95th percentile), not just the median: users remember slow requests. See latency budgets.
Splitting every metric
Compute every metric above split by prompt version, model, variant and category where you have them. Most changes in a metric line up with a change in one of these, and the split shows it immediately.