Wizard-of-Oz tests for AI features

Why this matters #

A prompt-first prototype tells you whether a model can do the task. It doesn’t tell you whether people want the result in the form you’d deliver it, at the moment you’d deliver it, with the controls you’d give them. Those are experience questions, and they’re often where AI features fail.

A Wizard-of-Oz test answers them before you build. Users interact with what looks like a working feature, while a person behind the scenes produces the responses. In a day, you learn what people ask for, what they expect, where they get confused and what they’d trust, and you get a set of ideal responses written by a human to aim for.

How a Wizard-of-Oz test works #

The name comes from the film’s wizard: an impressive machine that turns out to be a person behind a curtain. In product research, the “machine” is a realistic interface and the “wizard” is a team member writing the responses in real time.

1. 1 Participant acts

   Uses a realistic interface for a real task

2. 2 Request reaches the wizard

   Through a shared document, chat channel or simple admin screen

3. 3 Wizard responds

   Writes the response the ideal feature would give, following a short guide

4. 4 Participant reacts

   Reads, edits, trusts or ignores it; an observer takes notes

Show text descriptionHide text description

A sequence of 4 steps.

1. Participant acts : Uses a realistic interface for a real task
2. Request reaches the wizard : Through a shared document, chat channel or simple admin screen
3. Wizard responds : Writes the response the ideal feature would give, following a short guide
4. Participant reacts : Reads, edits, trusts or ignores it; an observer takes notes

Because a human writes the responses, you can test experiences the model can’t yet deliver reliably, then use what you learn to decide whether to build them, and what to build first.

Setting it up in a day #

You don’t need engineering for this. A typical setup:

• The interface. A clickable prototype in a design tool, a simple web page, or even a chat window, as long as it looks like the place the feature would live. Realism matters: users behave differently in a sketch than in something that feels like their product.
• The channel. Requests reach the wizard through anything fast: a shared document, a private chat channel, or a tiny admin page. Responses appear in the participant’s interface.
• The wizard’s guide. One page: what the feature does and doesn’t do, the tone, a few example responses, and rules for staying “in character”, such as how long a response should take and what to do with out-of-scope requests. Without a guide, the wizard becomes smarter than any real feature would be.
• Realistic timing. If the real feature would take five seconds, the wizard should too. An instant human response sets an expectation you can’t meet; a two-minute one tests patience, not the feature.
• An observer. Someone other than the wizard watches and takes notes, ideally with the participant thinking aloud.

Ethics matter. Get informed consent for the session and any recording. Tell participants at the end that a person was producing the responses, explain why, and give them the chance to withdraw their data. Don’t use real customer accounts or data in ways they haven’t agreed to. Many teams avoid the issue entirely by saying up front that “parts of this prototype are simulated”; that still gives useful results, because participants react to the experience itself.

What to watch for #

The responses are scripted; the participant’s reactions are the data. Watch for:

• What they ask for. The first requests reveal what people think the feature is for. They’re often different from what you designed.
• What they expect. Do they expect it to know things it couldn’t? To act, not just answer? To remember earlier sessions?
• What they change. Every edit to a response is a precise signal about what “good” means to them.
• When they trust and when they check. Which responses do they act on directly? Which do they verify? What makes them hesitate?
• Moments of confusion. Where do they pause, re-read, or ask the observer what to do?
• What they’d do without it. When the “AI” falls short, what’s their fallback? That’s your non-AI baseline.

Run five or so sessions, then look for patterns across them, as in user interviews.

From wizard to model #

A Wizard-of-Oz test leaves you with three valuable artifacts:

1. Real requests. What people actually asked, in their own words. These become inputs for your prompt-first tests and eval set.
2. Ideal responses. What the wizard wrote, and how participants edited it. These are reference answers to aim for and grade against.
3. Experience requirements. What participants needed in order to trust and use the output: sources, edit controls, timing, tone.

The next step is usually a prompt-first test using the real requests: can a model produce responses close to the wizard’s? Where it can, build. Where it can’t, you know precisely what’s missing and can decide whether to add context, tools or a human step, or to narrow the feature.

Common mistakes #

Try it #

Exercise

About 45 min

Plan a Wizard-of-Oz session for your feature

1. Choose the riskiest experience question for your feature, for example “do users trust drafted replies enough to send them?”
2. Describe the interface you’d show and how requests would reach the wizard.
3. Write a one-page wizard guide: scope, tone, response time, three example responses, and how to handle out-of-scope requests.
4. Write the consent and debrief script.

Show a hint

For the wizard guide, write the three example responses as if you were the best version of the feature you could realistically ship in three months, not the best possible assistant.

Show a worked solution

Example plan for drafted support replies:

• Question: do agents send drafts with light edits, or rewrite them?
• Interface: a copy of the helpdesk’s reply screen with a “Draft reply” button; drafts appear after about five seconds.
• Channel: requests are posted to a private chat channel; the wizard pastes the draft into a small admin page that shows it to the participant.
• Guide: use only help-center articles; under 120 words; friendly tone; if no article fits, reply “No help article covers this.”
• Consent and debrief: “Parts of this prototype are simulated. We’re testing the experience, not your performance.” After the session: “The drafts were written by a team member, so we could learn what a good draft needs before building it. You can ask us to delete your session notes.”

Check your understanding #

Key takeaways #