Adding LLM features to products: lessons from production

Adding LLM features to products

You've got a working product, users love it, and now everyone wants AI. The temptation is real: drop in an LLM API, add a chat box, ship it. But adding LLM features to products is nothing like adding a search bar or a new report. LLMs are probabilistic—they hallucinate, they're slow, they're expensive per request, and users notice when they're wrong in ways they forgive for traditional features.

The good news: if you treat LLM integration like a product feature (not magic), you can build something reliable that users actually want to use.

Start with a narrow, valuable use case

The worst LLM feature I've seen was a company that tried to use an LLM to solve every problem at once. Customer support, product recommendations, content generation, data classification—all in one sprint. Six weeks later, nothing shipped, the costs spiraled, and the team was burned out.

Instead, pick one problem. Make it specific enough that you can measure success. "Make our product smarter with AI" is not a use case. "Let users generate product descriptions from images and attributes" is.

Here's how I'd approach it:

1. Pick a manual, repetitive task your users already do
2. Estimate how many times per day it happens across your user base
3. Calculate if the time saved justifies the API cost
4. Check if failure modes are acceptable (can the user override or correct the output?)

Real example: I worked on a CRM where sales reps manually wrote follow-up email templates for each prospect. They did this 5-10 times a day, copying and pasting, personalizing. Perfect LLM use case. The cost was a few cents per email. If we saved them 2 minutes per email, at $100/hour burdened cost, the math worked. And if the LLM generated something weird, the rep just re-wrote it—no data loss.

A bad use case would have been: "Use LLM to auto-assign leads to sales reps." Why? Because a wrong assignment directly affects revenue, has no easy undo button, and the user experience of "why did this lead go to the wrong person?" is terrible.

Ground answers in your own data

Out-of-the-box LLMs are general-purpose. They know common facts, but they don't know your product, your business logic, or your user's context. The difference between a mediocre LLM feature and a great one is usually not a better model—it's better context.

If a user asks "What's my account balance?" and you send just that question to an LLM, you get a plausible-sounding but false answer. If you fetch their real balance from your database and include it in the prompt, you get a correct answer.

This pattern repeats everywhere:

• Search product catalog: Don't ask the LLM to list products. Retrieve matching products from your database, rank them, include their exact pricing and inventory, then let the LLM summarize or recommend.
• Generate personalized content: Include the user's actual history, preferences, and previous interactions. Don't ask the LLM to guess.
• Answer support questions: Include the user's account state, tickets, and knowledge base articles. The LLM becomes a summarizer and matcher, not an oracle.

Practically, this means building a retrieval layer before the LLM call:

async function generateEmailDraft(userId: string, prospectId: string) {
  // Fetch real data first
  const prospect = await db.prospects.findOne(prospectId);
  const userHistory = await db.interactions.findRecent(userId, prospectId, 5);
  const companyInfo = await db.companies.findOne(prospect.companyId);

  // Build a grounded prompt
  const prompt = `You are a sales assistant. Generate a 2-sentence follow-up email.

Prospect: ${prospect.name}
Company: ${companyInfo.name}
Industry: ${companyInfo.industry}
Last interaction: ${userHistory[0].summary}

Email:`;

  const response = await openai.createCompletion({
    model: "gpt-3.5-turbo",
    prompt,
    maxTokens: 150,
  });

  return response.choices[0].text;
}

The key difference: your database is the source of truth. The LLM is a transformer that makes it better for the user.

Measure quality with real examples

You can't scale what you don't measure. With LLMs, "works" is subjective. One user's perfect response is another user's waste of time.

Set up structured evaluation before you ship:

1. Define success criteria: For an email draft, maybe it's: includes prospect name, mentions a recent interaction, and is under 200 characters.
2. Create a test set: 20-50 real examples from your data. Not synthetic. Real.
3. Grade each output: Manual spot-check. Do it yourself or with a colleague. Score on a simple scale: unusable, needs edits, ship-ready.
4. Track cost and latency: How much did generating those 50 examples cost? How long did each take?

Document this in a simple table:

Aim for 70%+ "ship-ready" on your first pass. If you're at 40%, you either have the wrong prompt, the wrong model, or the wrong use case. Debug before shipping.

Plan for cost, latency and wrong answers

Every LLM call has three constraints: cost, latency, and correctness. You can't optimize all three. You have to choose, and that choice defines your UX.

Cost: A single API call to GPT-4 costs cents. At scale, this adds up. If your feature generates 1000 requests per day, and each costs $0.01, that's $10/day, $300/month. Is that worth $300/month in user value? If you have 100 paying customers, that's $3 per customer per month. Only you know if that's defensible.

Strategies to manage cost:

• Use cheaper models (GPT-3.5, Claude Haiku, open-source) for non-critical paths
• Batch requests at off-peak hours
• Cache responses for repeated queries
• Limit feature access to higher tiers or certain user cohorts

Latency: LLM APIs typically respond in 500ms to 3000ms. That's slow for a web request. Users expect sub-500ms responses for most interactions. Options:

• Make it async: show a spinner, send results via email or notification
• Pre-generate for common cases (user opens the feature, you've already started the LLM call)
• Use streaming: show tokens as they arrive, so it feels responsive
• Build a local cache of previous results

Wrong answers: This is the tricky one. Traditional features either work or fail. LLMs mostly work, but sometimes they confidently output garbage. Plan for it:

• Make it a draft, not final: Email draft, code suggestion, document outline—things the user edits. Never auto-publish or auto-execute.
• Show confidence signals: If the LLM isn't sure, say so. "This is a draft. Please review."
• Provide an easy undo or regenerate: User doesn't like the result? Click "Try again."
• Log everything for auditing: If something goes wrong, you need to know what the LLM saw and what it returned.

Here's a resilient pattern:

async function generateWithFallback(
  userId: string,
  input: string,
  fallbackFn: () => string
) {
  try {
    const result = await llmGenerate(input);
    
    // Simple quality check
    if (!result || result.length < 10) {
      console.warn("LLM output suspiciously short", { userId, input });
      return {
        text: fallbackFn(), // Use fallback
        source: "fallback",
        note: "Generated with template",
      };
    }

    return {
      text: result,
      source: "llm",
      note: "Draft generated by AI. Please review.",
    };
  } catch (error) {
    console.error("LLM call failed", { userId, error });
    return {
      text: fallbackFn(),
      source: "fallback",
      note: "Could not generate. Using template.",
    };
  }
}

Common mistakes

Shipping before measuring. You built a feature in a weekend, it feels smart, so you shipped it. Two weeks later, users complain it's slow, expensive, or wrong. Measure on real data first. Always.

No fallback. Your feature breaks, the LLM API is down, or the call times out. What does your user see? If it's a blank page or an error, you've made their experience worse. Have a sensible fallback: a template, a cached result, a note that this feature is temporarily unavailable.

Treating every request the same. Not all LLM requests are equal. A draft email can be slower and cheaper. A real-time chat response needs to be fast. Tailor your model choice and caching strategy to the use case.

No user control. The LLM output is a suggestion, not gospel. Let users edit, regenerate, and reject. This is both better UX and better for your data (users will correct the LLM, and you learn what doesn't work).

Forgetting the business model. Who pays for this feature? If it's you (you're paying the API cost), when does it pay for itself? If it's the user (you charge them per request), when does it stop being a gimmick and become a value-add?

Key takeaways

• Narrow scope wins. Pick one specific, repetitive task. Measure the impact. Prove the math before you scale.
• Ground in your data. Don't ask the LLM to guess. Build a retrieval layer that feeds it real information from your database.
• Measure on real examples. Create a test set of 20-50 examples from your actual data. Grade each output. Track cost and latency.
• Treat failure modes as features. Plan for slow responses (async), high costs (caching), and wrong answers (drafts, fallbacks, user override).
• Make it a draft, not a decision. The best LLM features are suggestions the user edits. Never auto-publish or auto-execute without explicit confirmation.
• Monitor and iterate. Log every LLM call. Review failures. Refine your prompt and grounding data based on real usage, not assumptions.