Skip to content
AI Product Engineer Wiki

    ↑ ↓ to move between results, Enter to open, Esc to close

    Menu

    AI feature launch checklist

    A pre-launch checklist for AI features covering evals, UX states, rollout, monitoring, cost, privacy and safety, with links to each topic.

    3 min read Updated
    On this page

    How to use this checklist

    Go through it before each stage of a rollout, not just once. Ticks are saved in this browser only. Every item links to the article that explains it. If an item doesn’t apply to your feature, write down why rather than skipping it silently.

    Problem and spec

    • The problem is backed by evidence, and the job is written down. (Problem discovery)
    • The one-page spec is current, including the non-AI alternative. (Spec)
    • The primary metric, guardrails and keep, change and remove thresholds are agreed. (Success metrics)

    Quality

    • A golden dataset of real cases exists, with categories and must-never-fail cases. (Golden datasets)
    • The launch threshold was agreed before measuring, and the feature meets it in every category. (Why evals)
    • Must-never-fail cases pass at 100%.
    • Any LLM judge is calibrated against human grades. (LLM-as-judge)
    • The eval suite runs in CI on prompt and model changes. (Evals in CI)

    Experience

    • Every state is designed and has been forced: waiting, streaming, ready, cut off, declined, nothing useful, busy, stopped, error. (States)
    • Output is editable and reversible where appropriate. (Editable outputs)
    • Factual answers show their sources. (Citations)
    • Risky actions require approval with a specific preview. (Approvals)
    • The entry point names the job, and limits are stated briefly. (Onboarding)
    • The non-AI path is always available.

    Code

    • The API key is server-side only; no secrets in client code or NEXT_PUBLIC_ variables. (Day-one demo)
    • Every stop reason is handled; tool calls never run from cut-off responses. (Model cheat sheet)
    • Model output is validated: schemas, formats, business rules and permissions. (Structured outputs)
    • Model names and prices come from one configuration, not scattered constants.

    Rollout

    • The feature, prompt version and model are behind flags. (Flags)
    • The staged rollout plan has exit criteria for each stage.
    • The kill switch has been tested, and the fallback is the non-AI path.
    • Analytics events record outcomes, prompt version, model and variant, without user content. (Analytics)
    • Traces record every request, including errors, with the request ID. (Tracing)
    • Alerts cover errors, refusals, truncation, latency, cost and volume, each with an owner.

    Safety and privacy

    • The feature has been threat-modeled for prompt injection, and the lethal trifecta is broken. (Prompt injection)
    • Guardrails make must-never-fail outcomes impossible, with false positives measured. (Guardrails)
    • The data flow is mapped, with retention set for every store and deletion that reaches every copy. (Privacy)
    • Users are told in plain language what happens to their data.

    Cost

    • Cost per success is estimated and compared with value. (Unit economics)
    • Rate limits and daily cost budgets are set per plan, enforced on the server. (Rate limiting)
    • Each API key has a spending limit; production, staging and CI use separate keys. (Cost dashboards)
    • Pricing covers heavy users, through limits or charges. (Pricing)

    After launch

    • A weekly review of traces and feedback is scheduled. (Capturing feedback)
    • The iteration loop and prompt change log are in place. (Iteration loop)
    • A monthly cost review is scheduled.
    • A date is set to compare results with the keep, change and remove thresholds.