Human-in-the-loop and approval flows
Design approval flows for AI actions, deciding which actions need a human, how to present a proposed action clearly, and how to keep approvals fast.
On this page
Why this matters
When an AI feature only produces text, a mistake costs a bad paragraph. When it can act, sending an email, issuing a refund, changing a setting, deleting a file, a mistake costs whatever that action costs. As features move from suggesting to doing, the approval step becomes the most important piece of UX you’ll design.
Good approval flows aren’t about asking permission for everything. That trains users to click “Approve” without reading, which is worse than no approval at all. They’re about asking at the right moments, showing exactly what will happen, and making the safe choice the easy one.
Sorting actions by risk
Start by listing every action your feature can take, then sort each by two questions: how much harm could a wrong action do, and can it be undone?
Show text descriptionHide text description
A two-by-two grid. The vertical axis is potential harm, increasing upwards. The horizontal axis is ease of undo, increasing to the right.
- top left (high potential harm, low ease of undo): Approve every time. Harmful and hard to undo: payments, sending external messages, deleting data. Preview and explicit confirmation.
- top right (high potential harm, high ease of undo): Approve, or apply with review. Harmful but reversible: changing permissions, bulk edits. Confirm, or apply and require review.
- bottom left (low potential harm, low ease of undo): Confirm in batches. Low harm but awkward to reverse: creating many records. Preview the batch once.
- bottom right (low potential harm, high ease of undo): Just do it, with undo. Low harm and reversible: drafts, tags, internal notes. Apply and show what changed.
Two refinements make this practical:
- Thresholds within an action. A $10 refund and a $5,000 refund are the same tool with very different risk. Set limits: below one value, approve quickly; above another, require a manager; above a hard cap, never through the AI.
- Hard limits in code. Some limits should never depend on the model or on a click: maximum amounts, protected records, actions on other users’ data. Enforce them in the tool’s code, so no prompt and no mistaken approval can exceed them. Guardrails covers this layer.
Presenting a proposed action
An approval is only as good as the information on screen. A useful approval shows:
- What will happen, in plain language and in specifics: “Refund $49.00 to the card ending 4242 for order A1042.”
- Why the AI proposes it: “The customer reported the item arrived damaged; the order was delivered 3 days ago.”
- What it affects: who’s notified, what changes, whether it can be undone.
- What it costs, when relevant: money, quota, credits.
- The choices: approve, edit before approving, or decline, with decline as easy to reach as approve.
Avoid approvals that only say “The assistant wants to use issue_refund. Allow?” Users can’t judge a tool name. They can judge a specific refund to a specific customer.
Approvals in code
Here’s the pattern with Claude’s tool use. The model can look up orders freely, but every refund pauses for an explicit human yes, and refunds above a hard limit are rejected in code no matter what. Inputs are validated before anything runs, and every way a response can end is handled.
import Anthropic from '@anthropic-ai/sdk';
import { createInterface } from 'node:readline/promises';
import { z } from 'zod';
const client = new Anthropic();
const terminal = createInterface({ input: process.stdin, output: process.stdout });
// Hard limit enforced in code: no prompt, model output or approval can exceed it.
const MAX_REFUND_USD = 200;
const tools: Anthropic.Tool[] = [
{
name: 'lookup_order',
description: 'Look up an order by its ID. Read-only.',
strict: true,
input_schema: {
type: 'object',
properties: { orderId: { type: 'string' } },
required: ['orderId'],
additionalProperties: false,
},
},
{
name: 'issue_refund',
description: 'Refund part or all of an order to the original payment method. A human must approve each refund.',
strict: true,
input_schema: {
type: 'object',
properties: { orderId: { type: 'string' }, amountUsd: { type: 'number' }, reason: { type: 'string' } },
required: ['orderId', 'amountUsd', 'reason'],
additionalProperties: false,
},
},
];
// Validate every tool input in code, even with strict schemas: never trust its shape.
const LookupInput = z.object({ orderId: z.string().min(1) });
const RefundInput = z.object({ orderId: z.string().min(1), amountUsd: z.number().positive(), reason: z.string().min(1) });
// Stand-ins for your real systems.
async function lookupOrder(orderId: string) {
return { orderId, totalUsd: 49, status: 'delivered', deliveredDaysAgo: 3 };
}
async function issueRefund(orderId: string, amountUsd: number) {
return { refundId: `rf_${orderId}`, amountUsd };
}
async function runTool(block: Anthropic.ToolUseBlock): Promise<Anthropic.ToolResultBlockParam> {
const ok = (content: unknown): Anthropic.ToolResultBlockParam => ({
type: 'tool_result',
tool_use_id: block.id,
content: JSON.stringify(content),
});
const fail = (content: string): Anthropic.ToolResultBlockParam => ({
type: 'tool_result',
tool_use_id: block.id,
content,
is_error: true,
});
switch (block.name) {
case 'lookup_order': {
const input = LookupInput.safeParse(block.input);
return input.success ? ok(await lookupOrder(input.data.orderId)) : fail('Invalid input for lookup_order.');
}
case 'issue_refund': {
const input = RefundInput.safeParse(block.input);
if (!input.success) return fail('Invalid input for issue_refund.');
const { orderId, amountUsd, reason } = input.data;
if (amountUsd > MAX_REFUND_USD) {
return fail(`Refunds above $${MAX_REFUND_USD} need a manager. Tell the customer it has been escalated.`);
}
// Show exactly what will happen, then require an explicit yes. Anything else is a no.
const answer = await terminal.question(
`\nRefund $${amountUsd.toFixed(2)} for order ${orderId}?\nReason given: ${reason}\nApprove? [y/N] `,
);
if (answer.trim().toLowerCase() !== 'y') {
return fail('A human declined this refund. Do not retry it; explain the outcome to the customer.');
}
return ok(await issueRefund(orderId, amountUsd));
}
default:
return fail(`Unknown tool: ${block.name}`);
}
}
const textOf = (message: Anthropic.Message) =>
message.content.flatMap((b) => (b.type === 'text' ? [b.text] : [])).join('');
async function handle(customerMessage: string): Promise<void> {
const messages: Anthropic.MessageParam[] = [{ role: 'user', content: customerMessage }];
// A turn limit stops a confused loop from running (and costing) forever.
for (let turn = 0; turn < 10; turn++) {
const response = await client.messages.create({
model: 'claude-opus-5-5',
max_tokens: 4096,
output_config: { effort: 'medium' },
system: 'You are a support agent. Look up orders before acting. Propose refunds only when the policy clearly allows.',
tools,
messages,
});
messages.push({ role: 'assistant', content: response.content });
switch (response.stop_reason) {
case 'end_turn':
case 'stop_sequence':
console.log(`\n${textOf(response)}`);
return;
case 'tool_use': {
// Run tools one by one so approvals appear in order, then return all results in one message.
const results: Anthropic.ToolResultBlockParam[] = [];
for (const block of response.content) {
if (block.type === 'tool_use') results.push(await runTool(block));
}
messages.push({ role: 'user', content: results });
break;
}
case 'pause_turn':
// The response is already in the history; sending it back lets the model continue.
break;
case 'max_tokens':
case 'model_context_window_exceeded':
// Never run a tool call from a cut-off response: its input may be incomplete.
console.log('\nThe response was cut off before it finished. No action was taken.');
return;
case 'refusal':
console.log('\nThis request was declined. Please handle it manually.');
return;
case null:
console.log(`\nUnexpected stop (${response.stop_reason}). No further action taken.`);
return;
}
}
console.log('\nStopped after 10 turns without finishing. Escalating to a human.');
}
await handle('Hi, order A1042 arrived damaged. Can I get my money back?');
terminal.close(); Run it with npx tsx agents/support-agent.ts. Things to notice:
- The approval lives in the tool’s code, not in the prompt. The model can’t skip it, and a declined approval goes back to the model as a clear result it can explain to the customer.
- The hard limit sits outside the approval. Even a mistaken “y” can’t refund more than the cap.
- Inputs are validated before anything runs, and invalid input becomes an error result rather than a crash.
- Cut-off responses never run tools. A
max_tokensstop could leave a tool call half-written; the loop stops instead. - There’s a turn limit, so a confused loop ends and escalates instead of running indefinitely.
In a real product the terminal prompt becomes an approval card in your interface, and the loop pauses until the user responds. The SDK’s tool runner (opens in a new tab) can drive this loop for you too; the approval check stays inside the tool. Project 3 builds a full version with an interface.
Avoiding approval fatigue
If users approve dozens of things an hour, they stop reading. Ways to keep approvals meaningful:
- Only ask when it matters. Move low-risk, reversible actions to “just do it, with undo”.
- Batch related actions. “Create these 12 tasks?” once, instead of 12 separate approvals.
- Let users raise trust deliberately. “Always allow tagging” is fine; “always allow refunds” should be restricted or impossible.
- Make declining as easy as approving. Same size, same prominence. Never pre-select approve on irreversible actions.
- Learn from declines. Frequent declines on one kind of action mean the AI proposes it too often; fix the prompt or the policy, and add those cases to your evals.
Common mistakes
- Asking for approval of everything. Users stop reading. Reserve approvals for harmful or irreversible actions.
- Approvals users can’t judge. “Allow issue_refund?” means nothing. Show the specific action, its target, its effect and its cost.
- Relying on the prompt for safety. “Always ask before refunding” in a prompt can be ignored or manipulated. Put approvals and limits in the tool’s code.
- Running tool calls from cut-off responses. Check the stop reason; only run tools when the response ended with a complete tool call.
- No turn limit on agent loops. Cap the number of turns and escalate when it’s reached.
Try it
Exercise
About 30 min
Classify the actions of an agent by risk
- List every action an AI feature or agent in your product could take, or might take in future.
- Place each one in the matrix above.
- For the top-left quadrant, write the exact text of the approval card, including amounts, targets and consequences.
- Identify at least one hard limit to enforce in code, regardless of approval.
Show a hint
Look for actions that touch money, external people (emails, messages, invitations), permissions or deletion. Those almost always belong in the top half of the matrix.
Show a worked solution
Example for a project-management assistant:
- Just do it, with undo: add labels, write internal notes, draft task descriptions.
- Confirm in batches: create tasks from meeting notes.
- Approve, or apply with review: reassign tasks to other people, change due dates on others’ tasks.
- Approve every time: invite external guests, delete projects, send messages to clients.
- Approval card: “Invite jane@client.com as a guest to Website redesign? She’ll see all tasks and files in this project. You can remove her later.”
- Hard limit: the assistant can never delete projects with more than 10 tasks, and can never change billing.
Check your understanding
Key takeaways
- Sort actions by potential harm and ease of undo before designing the flow.
- Approvals must show the specific action, its reason, its effect and its cost, with decline as easy as approve.
- Put approvals and hard limits in code, validate tool inputs, never act on cut-off responses, and cap agent turns.
- Avoid fatigue: ask only when it matters, batch, and expand autonomy with evidence.
Progress is saved in this browser and lights up the roadmap.
Related articles
- Guardrails for inputs, outputs and actions Add guardrails around AI features with input checks, output validation, action limits and refusal handling, and keep them from blocking real users. Level: Expert
- Prompt injection and tool-use risks Understand prompt injection and the risks of giving AI features tools, including attack paths, realistic threat models and defenses that limit the damage. Level: Expert
- Streaming and loading states Design streaming responses and loading states for AI features, from when to stream to skeletons, progress, stop buttons and handling a stream that ends early. Level: Intermediate