Writing a product case study
Write a case study of an AI feature that shows how you think, covering context, problem, decisions and trade-offs, evals, results and what you would change.
On this page
Why this matters
Code shows that you can build. A case study shows how you think: which problem you chose and why, the options you considered, the trade-offs you accepted, how you knew it worked, and what you’d do differently. That’s exactly what hiring managers and interviewers are trying to learn about you, and it’s very hard to show any other way.
A good case study is also useful beyond job hunting: it’s how you explain a project to your team, your manager or your future self. This article gives you a structure, shows how to write about trade-offs and numbers honestly, and covers how to edit it for busy readers.
The structure
A case study tells the story of a decision-making process, not a list of features. This structure works for almost any AI product project:
Show text descriptionHide text description
A sequence of 6 steps.
- Summary : Problem, what you did, the result: three or four sentences
- Context and problem : Who, what job, the evidence it mattered
- Approach : What you built and why that shape
- Key decisions : Two or three trade-offs, with the options you rejected
- Results : Evals, product metrics, cost, honestly reported
- Lessons : What didn't work, and what you'd do next
Summary. Three or four sentences a reader could stop after: “Support agents spent hours on repetitive how-to tickets. I built AI-drafted replies grounded in our help center, with an eval set of 100 real tickets and a staged rollout. Drafts reached 46% acceptance and cut median reply time on those tickets by about a fifth, at roughly three cents per accepted draft.”
Context and problem. Who had the problem, how often, how you know. Keep it short and concrete, with evidence: ticket counts, interview quotes, metrics.
Approach. What you built, at the level a product person would describe it. Include one diagram or screenshot. Mention the technology only where it explains a decision.
Key decisions. The heart of the case study. See the next section.
Results. Eval results, product metrics, cost. Before and after where you can.
Lessons. What didn’t work, what surprised you, what you’d do next. This section separates experienced writers from beginners.
Showing trade-offs
Interesting case studies are built on decisions where the right answer wasn’t obvious. For each of your two or three key decisions, write:
- The decision: “Draft replies for agents to edit, rather than sending them automatically.”
- The options: automatic sending for simple tickets; drafts for all; no AI at all, only better saved replies.
- What you chose and why: “Drafts for how-to tickets only. A wrong automatic reply to a customer is expensive and hard to undo; agents catch mistakes in seconds, and every edit gives us feedback.”
- What it cost you: “Slower than automatic sending, and the time savings depend on agents trusting the drafts.”
- How it turned out: “Acceptance rose from 31% to 46% over three prompt iterations; we never needed automatic sending for the volume we had.”
The rejected options and the honest cost are what make this convincing. A decision with no alternatives isn’t a decision, and a choice with no downside reads as hindsight.
Numbers that matter
Numbers make a case study credible, and misused numbers destroy credibility fast. Guidelines:
- Report what you measured, how, and on how much data. “88% pass rate on a 100-case eval set built from real tickets, graded by two senior agents” is credible. “88% accuracy” is not.
- Show before and after for changes you made, measured the same way.
- Include cost per success, not just quality. It shows you think about the product as a business.
- Round sensibly and don’t overclaim. “About a fifth faster” from a two-week test is honest; “21.4% faster” implies precision you don’t have.
- Separate correlation from causation. If you didn’t run an A/B test, say the change coincided with an improvement rather than caused it.
- Respect confidentiality. For work projects, get permission and use relative numbers if absolute ones are sensitive.
Editing for readers
Most readers will read the summary, glance at the headings and look at one or two sections. Edit for that:
- Length: roughly 800 to 1,500 words for the full case study, with a summary that stands alone.
- Headings that say something: “Why drafts, not automatic replies” beats “Decision 1”.
- One or two visuals: a screenshot of the feature, and maybe a simple chart of a metric over time.
- Plain language: explain any jargon a product manager wouldn’t know.
- A clear role statement: “I led the design and evals; a teammate built the helpdesk integration.”
Then ask someone outside the project to read only the summary and tell you what you did and what happened. If they can’t, rewrite the summary.
Common mistakes
- A feature list instead of a story. Describe the problem, decisions and results, not every capability.
- No rejected options. Without alternatives, decisions look obvious or post hoc. Show what you didn’t choose and why.
- Vague or precise-looking numbers without context. Say what you measured, how and on how much data.
- Burying the result. The summary at the top should tell the whole story.
- Skipping the lessons. Honest reflection on what didn’t work is a sign of experience.
Try it
Exercise
About 90 min
Write a one-page case study draft
- Pick a project: one from work (with permission), a side project or one of this wiki’s projects.
- Write the three- or four-sentence summary first.
- Write the key decisions section with at least two decisions, each with options, choice, cost and outcome.
- Add results with sources and sample sizes, and a lessons section. Ask someone to read only the summary and play it back to you.
Show a hint
If you’re stuck on decisions, look at your change log and spec updates. Every “because” you wrote down there is a decision worth explaining.
Show a worked solution
A strong draft has:
- A summary that names the problem, what you did and the measured result in under 80 words.
- Two decisions with real alternatives and an honest downside each.
- Results with how they were measured: eval set size and source, metric definitions, test duration, cost per success.
- A lessons section with at least one thing that didn’t work and one thing you’d do differently.
- A role statement that makes your contribution clear.
Check your understanding
Key takeaways
- A case study shows how you think: problem, approach, decisions, results, lessons.
- Lead with a summary that tells the whole story.
- Build it on two or three real trade-offs, with rejected options and honest costs.
- Report numbers with how they were measured, separate correlation from causation, and reflect on what didn’t work.
Progress is saved in this browser and lights up the roadmap.
Related articles
- The iteration loop: from feedback to prompt changes Turn production feedback into measured improvements by finding failure patterns, changing prompts or design, verifying with evals and shipping safely. Level: Advanced
- Career paths and leveling Plan your career as an AI product engineer, including what changes from mid-level to senior and staff, adjacent paths, and how to grow scope and impact. Level: Expert