Feature flags and staged rollouts for AI

Why this matters #

However good your eval set is, production will show you inputs it didn’t contain: new phrasings, unexpected languages, documents in odd formats, usage patterns nobody predicted. AI features meet more of these surprises than most, because their inputs are open-ended and their outputs vary.

A staged rollout limits the damage while you learn. Instead of shipping to everyone at once, you expose the feature to a growing share of users, watch quality, behavior and cost at each step, and keep a switch that turns it off in seconds. Feature flags are the mechanism. This article covers how to flag AI features, plan a rollout with exit criteria, design kill switches and fallbacks, and implement it in code.

Flags for AI features #

A conventional flag answers one question: is the feature on for this user? AI features benefit from flagging three things separately:

• The feature itself: on or off, for whom. Your kill switch lives here.
• The prompt version: so you can roll a new prompt out gradually, and back, without a deploy.
• The model and its settings: so a model upgrade or effort change goes through the same staged process as any other change.

Treating prompts and models as flagged configuration, not hard-coded constants, is what makes AI features safe to change. A prompt edit is a behavior change for every user; it deserves the same rollout discipline as code. And because flags live outside the deployment, a bad change can be reverted in seconds instead of waiting for a build.

A staged rollout plan #

Write the plan before the first user sees the feature. Each stage has an audience, a minimum duration and exit criteria that must hold before you move on:

Example exit criteria for each stage:

Spend at least long enough at each stage to see normal weekly patterns in your traffic, and don’t skip stages because things look fine on day one. Problems with rare inputs only show up with enough volume.

Kill switches and fallbacks #

A kill switch turns the AI feature off for everyone, immediately, without a deploy. It should be:

• Fast: a flag change takes effect in seconds or minutes.
• Obvious: everyone on call knows where it is and is allowed to use it.
• Tested: flip it in staging, and once in production during a quiet period, before you need it.

What users see when the feature is off matters as much. Plan a fallback for each AI feature:

• Hide the entry point, so nothing looks broken.
• Fall back to the non-AI path: the manual form, the saved replies, the plain search. This is where the non-AI baseline pays off again.
• Fall back to the previous prompt or model if the problem is a recent change rather than the feature itself.

Flags in code #

Here’s a small, self-contained implementation of the ideas above. Users are assigned to a bucket from 0 to 99 by hashing their ID together with the flag name, so each user always gets the same answer, on every server, without storing anything. One flag controls the feature, its prompt version and its model.

import { createHash } from 'node:crypto';

/** One flag controls the feature, the prompt version and the model, so each can change without a deploy. */
interface AiFeatureFlag {
  /** Kill switch: false turns the feature off for everyone. */
  enabled: boolean;
  /** Share of users who get the feature, from 0 to 100. */
  rolloutPercent: number;
  promptVersion: string;
  model: string;
  /** Always included, regardless of the percentage: the team and early testers. */
  allowlist: string[];
}

// In production, load these from your flag service or config store so they change without deploying.
const flags: Record<string, AiFeatureFlag> = {
  'reply-drafts': {
    enabled: true,
    rolloutPercent: 10,
    promptVersion: 'reply-v4',
    model: 'claude-opus-5-5',
    allowlist: ['user-internal-1'],
  },
};

/** Stable bucket from 0 to 99: the same user and flag always land in the same bucket. */
export function bucket(userId: string, flagKey: string): number {
  return createHash('sha256').update(`${flagKey}:${userId}`).digest().readUInt32BE(0) % 100;
}

export type Decision =
  | { on: true; promptVersion: string; model: string }
  | { on: false; reason: 'unknown-flag' | 'disabled' | 'not-in-rollout' };

export function decide(flagKey: string, userId: string): Decision {
  const flag = flags[flagKey];
  if (!flag) return { on: false, reason: 'unknown-flag' };
  if (!flag.enabled) return { on: false, reason: 'disabled' };
  const included = flag.allowlist.includes(userId) || bucket(userId, flagKey) < flag.rolloutPercent;
  return included ? { on: true, promptVersion: flag.promptVersion, model: flag.model } : { on: false, reason: 'not-in-rollout' };
}

// Usage: decide once per request, and record the decision with your analytics and traces.
console.log(decide('reply-drafts', 'user-internal-1'));

// Sanity check: about 10% of simulated users should be included.
let included = 0;
for (let i = 0; i < 10_000; i++) if (decide('reply-drafts', `user-${i}`).on) included++;
console.log(`${(included / 100).toFixed(1)}% of 10,000 simulated users included`);

Properties worth noticing:

• Raising the percentage only adds users. A user in bucket 7 is included at 10% and stays included at 50%, so nobody loses the feature mid-rollout.
• Different flags bucket independently, because the flag name is part of the hash. The same users aren’t always first in line for every experiment.
• The decision includes the prompt version and model, which you should log with every request. Without that, you can’t tell which configuration produced which outcome.
• The reason for “off” is explicit, which helps when debugging why a user doesn’t see the feature.

In a real product, use your existing feature flag service if you have one; most support percentage rollouts and allowlists. The important part is the discipline: prompts and models behind flags, decisions logged, and a tested kill switch.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #