Project 2: Smart catalog search with RAG and citations

What you’ll build #

A search experience where shoppers ask questions in their own words, such as “a waterproof jacket for cycling under $150” or “which of these blenders can crush ice?”, and get a short answer that cites the specific products it’s based on, with links. Behind it: a retrieval step that finds candidate products, a model that answers only from those products, and an eval set that measures whether the right products are found and whether answers are correct.

The headline result you’re aiming for is a comparison: on the same set of real queries, does your AI search find relevant products more often than the existing keyword search, and where does it do worse?

Before you start #

• Context windows, retrieval and memory: when to retrieve versus send everything
• Citations and showing sources: the answer function you’ll reuse
• Building golden datasets: for the query set
• When not to use AI: measuring the baseline first

You’ll need a product catalog: a real one if you can get permission (a friend’s shop, an open dataset), or a realistic one of at least 200 products with names, descriptions, attributes and prices.

What you’ll practice #

• Retrieval: finding the few products that matter for a query
• Citations: answers grounded in, and linked to, specific products
• Golden datasets: real queries with known relevant products
• Relevance evals: measuring retrieval and answers separately

Milestones #

1. Measure the keyword-search baseline #

Collect 50 real or realistic shopper queries: from site search logs if you have them, otherwise write them with someone who shops in that category, including vague ones (“gift for a runner”), specific ones (“size 10 trail shoes”) and ones the catalog can’t answer. For each, list the products a knowledgeable person would consider relevant. Run them through the existing search (or a simple keyword match) and record which relevant products appear in the top five results.

2. Index the catalog for retrieval #

Turn each product into a searchable text: name, category, key attributes, description, price. Start simple. The retriever below scores products by how many query terms they contain, weighting rarer terms more heavily, and needs no external services. Later you can replace it with embeddings or your database’s full-text search, and compare.

export interface Product {
  id: string;
  name: string;
  text: string; // Name, category, attributes, description and price, joined.
}

const tokenize = (s: string) => s.toLowerCase().match(/[a-z0-9]+/g) ?? [];

/** Builds a scorer that ranks products by matching query terms, weighting rare terms more (a simple TF-IDF). */
export function buildIndex(products: Product[]) {
  const docs = products.map((p) => ({ product: p, terms: tokenize(p.text) }));
  const docFreq = new Map<string, number>();
  for (const d of docs) for (const t of new Set(d.terms)) docFreq.set(t, (docFreq.get(t) ?? 0) + 1);
  const idf = (t: string) => Math.log((docs.length + 1) / ((docFreq.get(t) ?? 0) + 1)) + 1;

  return function search(query: string, k = 5): Product[] {
    const queryTerms = [...new Set(tokenize(query))];
    return docs
      .map((d) => ({
        product: d.product,
        score: queryTerms.reduce((s, t) => s + (d.terms.filter((x) => x === t).length > 0 ? idf(t) : 0), 0),
      }))
      .filter((r) => r.score > 0)
      .sort((a, b) => b.score - a.score)
      .slice(0, k)
      .map((r) => r.product);
  };
}

/** Share of a query's relevant products that appear in the top k results, averaged over queries. */
export function recallAtK(results: Map<string, string[]>, relevant: Map<string, string[]>): number {
  let total = 0;
  for (const [query, wanted] of relevant) {
    const found = results.get(query) ?? [];
    total += wanted.length === 0 ? 1 : wanted.filter((id) => found.includes(id)).length / wanted.length;
  }
  return total / Math.max(1, relevant.size);
}

const products: Product[] = [
  { id: 'p1', name: 'Storm Shell Jacket', text: 'Storm Shell Jacket waterproof breathable cycling jacket reflective $129' },
  { id: 'p2', name: 'Trail Runner 4', text: 'Trail Runner 4 running shoes trail grip lightweight $110' },
  { id: 'p3', name: 'City Parka', text: 'City Parka insulated winter coat water resistant $180' },
  { id: 'p4', name: 'Aero Cycling Vest', text: 'Aero Cycling Vest windproof cycling vest packable $65' },
];

const search = buildIndex(products);
const queries = new Map([
  ['waterproof jacket for cycling', ['p1']],
  ['shoes for trail running', ['p2']],
  ['warm coat for winter commuting', ['p3']],
  ['sneakers for mountain trails', ['p2']], // No shared words: the catalog says "running shoes" and "trail".
]);
const results = new Map([...queries.keys()].map((q) => [q, search(q, 2).map((p) => p.id)]));
for (const [q, ids] of results) console.log(`${q} → ${ids.join(', ')}`);
console.log(`Recall@2: ${(recallAtK(results, queries) * 100).toFixed(0)}%`); // 75%: the synonym query finds nothing.

Measure recall@5 on your query set (the share of relevant products found in the top five) before moving on. If retrieval doesn’t find the right products, no prompt will produce the right answer. Keyword retrieval struggles with synonyms (“coat” versus “parka”, “commuting” versus “city”): that’s exactly what your comparison should reveal, and what embeddings or query rewriting can fix later.

3. Generate answers with citations #

For each query, pass the top retrieved products as documents to the answer-with-citations function, with a system prompt that says to recommend only from the provided products and to say clearly when none fit. Render the answer with product cards for each citation, linking to the product page.

4. Build a 50-query relevance eval set #

Turn your milestone 1 queries into a golden dataset with two kinds of checks:

• Retrieval: recall@5 against the relevant products you listed.
• Answers: for each answer, does it recommend only relevant products, cite them, respect constraints like price and size, and say “nothing fits” for the unanswerable queries? Grade a sample yourself, then consider an LLM judge for the rest.

5. Compare against the baseline and report #

Put the results side by side: keyword search recall@5 versus your retrieval, answer correctness, and cost per query. Break results down by query type: vague, specific, constraint-heavy, unanswerable. Expect a mixed picture, with AI search winning on vague queries and possibly losing on exact product codes, and design a hybrid if so.

Done when #

• Shoppers can ask natural-language questions and get short answers citing specific products with links.
• Answers stay within the retrieved products and say clearly when nothing fits.
• A 50-query eval set measures retrieval (recall@5) and answer quality separately.
• You have a written comparison against keyword search, broken down by query type, with cost per query.

Write it up #

Lead with the comparison: where AI search beats keyword search, where it doesn’t, and what you did about it. Include the recall numbers, a screenshot of a cited answer and of a “nothing fits” answer, and the decision you made about a hybrid. Explain one retrieval failure in detail, which shows you understand the system end to end.

Stretch goals #

• Replace keyword retrieval with embeddings or hybrid search and compare recall@5.
• Add query rewriting: have a small model turn vague queries into structured filters before retrieval.
• Cache the catalog prompt and measure the cost difference.