Project 1: AI writing assistant inside an existing app

What you’ll build #

A writing assistant inside an app you already have, or a simple one you create: a notes app, a support inbox, a CMS, a project tracker. When the user is writing, they can ask for a suggestion (a draft, a rewrite, a shorter version), see it stream in, edit it, and accept, regenerate or discard it. Every suggestion records what happened to it, so at the end you can say how many suggestions people kept and how much they changed them.

It’s deliberately small. The point isn’t the AI call, which takes an hour. It’s the parts around it that make a feature people trust and that you can measure.

Before you start #

You’ll use these articles directly:

• Prompt-first prototyping, for the first milestone
• Streaming and loading states, for the route and component
• Editable outputs and undo, for accept, reject and the edit ratio
• Product analytics events, for the event schema

You’ll need a Next.js app (or another web framework you know) and an Anthropic API key with a spending limit set in the Console.

What you’ll practice #

• Prompting on real drafts, with versions and a review sheet
• Streaming UI with every final state designed
• Editable outputs that keep the user in control
• Feedback events that turn usage into a quality metric

Milestones #

1. Pick the app and the writing job #

Choose one specific writing job, not “help with writing”. For example: “turn rough meeting notes into a short summary for the team”, or “rewrite a support reply to be friendlier and shorter”. Write it as a job statement and define your primary metric: probably the share of suggestions accepted with light or no edits.

2. Prompt-first test on real drafts #

Collect 15 to 20 real examples of the writing (your own notes, anonymized replies, public examples). Run the prompt harness over them, review every output, and iterate until at least three quarters would be usable with light edits. Keep the inputs: they’re your first eval set.

3. Streaming suggestion UI with accept, edit and reject #

Build the route and component from the streaming article, adapted to your writing job. The suggestion appears in an editable area with Accept, Regenerate and Discard. Accept inserts the edited text and shows Undo for a few seconds. Design and force every final state: done, cut off, declined, stopped and error.

4. Log acceptance and edit events #

Send ai_requested, ai_completed, ai_accepted (with the edit ratio), ai_retried and ai_dismissed events with the prompt version. Keep user text out of analytics. Check in your browser’s network panel that each action sends exactly one outcome event.

5. Review a week of usage #

Use the assistant yourself for real work for a week, or give it to two or three friends or colleagues. Then compute acceptance rate, average edit ratio and retry rate, read the most heavily edited suggestions, and make one prompt change based on what you find. Measure again.

Done when #

• The assistant handles a real writing job in a real or realistic app.
• Suggestions stream, are editable, and can be accepted, regenerated, discarded and undone.
• Every final state is designed and has been forced at least once.
• Events record outcome, edit ratio and prompt version, with no user text.
• You have at least a week of usage, metrics before and after one prompt change, and notes on the most-edited suggestions.

Write it up #

A one-page case study following writing a product case study: the writing job, the prompt iterations and what drove them, a screenshot of the editable suggestion (and one of an error state), and your metrics before and after the prompt change. End with what you’d change next.

Stretch goals #

• Show the AI’s original suggestion and the user’s final version side by side in an internal review page, to speed up your weekly review.
• Add a thumbs control with a short reason list and compare it with the edit ratio.
• Try lower effort on the model and compare latency, cost and acceptance.