Guardrails for inputs, outputs and actions

Why this matters #

Models are probabilistic: even a well-prompted, well-evaluated feature will occasionally produce something it shouldn’t, such as a refund amount that isn’t in the policy, a link to a site you don’t control, or a promise your business can’t keep. Prompts reduce how often that happens; they can’t make it impossible.

Guardrails are the deterministic layer around the model that makes the worst outcomes impossible rather than unlikely. Good ones are narrow, specific to your product’s real risks, and measured so they don’t block legitimate users. This article covers the three layers of guardrails, how to validate output and limit actions in code, and how to keep false positives in check.

Three layers of guardrails #

Input guardrails stop problems before they reach the model: reject empty or enormous inputs, check that requests fall within the feature’s purpose, and mark untrusted content as data, as covered in prompt injection.

Output guardrails check what the model produced before anyone sees it or acts on it.

Action guardrails sit between the model’s decision and anything that changes the world: permissions, hard limits and approvals.

The model layer in the middle, with good prompts, structured outputs and the provider’s own safety measures, does most of the work. The other layers exist for the cases where it doesn’t.

Validating output #

The most valuable output guardrails check the specific claims that would hurt most if wrong. For a support-reply feature, that means money amounts, links and promises. Here’s a guard that checks a drafted reply against the policy before it’s shown:

interface Policy {
  /** Amounts the business actually offers, as written in the policy. */
  allowedAmounts: string[];
  /** Hosts replies may link to. */
  allowedHosts: string[];
}

export interface GuardResult {
  ok: boolean;
  violations: string[];
}

// Promises support can't keep unless the policy explicitly allows them.
const forbiddenPromises = [/\bguarantee[ds]?\b/i, /\bwithin (24|48) hours\b/i, /\bfull refund\b/i];

/** Checks a drafted reply against the policy. Anything it flags goes to human review instead of the customer. */
export function guardReply(reply: string, policy: Policy): GuardResult {
  const violations: string[] = [];

  for (const amount of reply.match(/[$€£]\s?\d+(?:[.,]\d{2})?/g) ?? []) {
    const normalized = amount.replace(/\s/g, '');
    if (!policy.allowedAmounts.includes(normalized)) violations.push(`amount not in policy: ${normalized}`);
  }

  for (const url of reply.match(/https?:\/\/[^\s)]+/g) ?? []) {
    let host = '';
    try {
      host = new URL(url).hostname;
    } catch {
      // Unparseable URLs fall through to the check below as unknown hosts.
    }
    if (!policy.allowedHosts.some((h) => host === h || host.endsWith(`.${h}`))) violations.push(`link to unknown host: ${url}`);
  }

  for (const pattern of forbiddenPromises) {
    const match = reply.match(pattern);
    if (match) violations.push(`promise needs approval: "${match[0]}"`);
  }

  return { ok: violations.length === 0, violations };
}

const policy: Policy = { allowedAmounts: ['$15', '$25'], allowedHosts: ['help.example.com'] };

console.log(guardReply('Sorry about the delay! We have added a $15 credit. Details: https://help.example.com/credits', policy));
console.log(guardReply('We guarantee a full refund of $80 within 24 hours. See https://refund-help.net/claim', policy));

Design choices worth copying:

• Check the specific, high-risk claims (amounts, links, promises), not “quality” in general. Narrow checks are reliable; broad ones aren’t.
• Compare against the source of truth, the policy, rather than against what seems plausible.
• Flag, don’t silently fix. A flagged reply goes to a human review state with the reasons shown. Rewriting it automatically would hide the problem and might introduce new errors.
• Keep the logic in code you can test. Every guard should have unit tests with passing and failing examples.

Other output checks that are often worth having: schema validation for structured output, required disclaimers in regulated contexts, and detection of personal data from other users appearing in a response.

Limiting actions #

Action guardrails apply the same principle to tools: the model proposes, code disposes. Before any tool with side effects runs, code checks:

• Permissions: can this user perform this action on this object? Check with the same rules as the rest of your app, using the signed-in user’s identity, never an identity the model supplied.
• Hard limits: amounts, counts and scopes that can’t be exceeded regardless of what the model or user says.
• Approval thresholds: when a person has to confirm, with a specific preview.
• Rate and budget limits on the action itself, as in abuse and rate limiting.

The approval flow example shows a refund tool with all of these: validated input, a hard cap, and an explicit approval.

Measuring false positives #

A guardrail that blocks legitimate use is a bug with a safety label. Users who keep hitting it lose trust in the feature, and teams start adding exceptions until the guardrail protects nothing. Measure both directions:

• Catch rate: of the outputs that should be blocked, how many are? Test with eval cases built from real failures and must-never-fail cases.
• False positive rate: of the outputs that are fine, how many are wrongly flagged? Sample flagged outputs weekly and review them.

Log every guardrail decision with the rule that fired. Then you can see which rules flag most often, review a sample, and tighten the ones that misfire. In the example above, if "within 24 hours" keeps flagging replies that legitimately describe a policy that does promise 24-hour responses, the fix is to add that case to the policy, not to delete the check.

Common mistakes #

Try it #

Check your understanding #

Key takeaways #