Introduction
We've all seen it: a shiny new LLM feature launches to production, performs beautifully in demos, then silently fails under real traffic. After shipping AI features across multiple production systems—e-commerce platforms processing thousands of orders, CRM systems handling customer interactions, real-time trading dashboards—we've learned that common LLM integration mistakes are less about the models themselves and far more about how we integrate them into our applications. The gap between a working prototype and a reliable production system comes down to validation, resilience, and boundaries.
This article walks through the mistakes we've made and seen others make, with concrete examples and working solutions. Whether you're adding GPT-backed search to a marketplace or building AI-assisted workflows for a SaaS platform, these lessons will save you production incidents.
Trusting model output without validation
The first instinct when calling an LLM is to assume the response is correct. It rarely is. Models hallucinate facts, return malformed data, contradict themselves, and occasionally produce content that violates your business rules entirely.
We integrated Claude into an e-commerce recommendation engine. The model was asked to rank products by relevance to a customer query. Most responses looked perfect—clean JSON with product IDs and scores. Then one day, it returned product IDs that didn't exist in our catalog. Our frontend crashed trying to fetch metadata. The model had fabricated IDs.
Here's what we should have done from the start:
import json
from typing import Optional
def get_product_recommendations(query: str, valid_product_ids: set) -> Optional[list]:
"""
Get LLM recommendations with validation.
"""
prompt = f"""Recommend 5 products for this customer query: {query}
Return JSON with format: {{"recommendations": [{{"product_id": int, "score": float}}]}}"""
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=500,
messages=[{"role": "user", "content": prompt}]
)
try:
# Parse response
content = response.content[0].text
data = json.loads(content)
# Validate structure
if "recommendations" not in data or not isinstance(data["recommendations"], list):
logger.error("Invalid response structure", extra={"response": content})
return None
# Validate each recommendation
validated = []
for rec in data["recommendations"]:
if not isinstance(rec, dict):
logger.warning("Skipping non-dict recommendation")
continue
product_id = rec.get("product_id")
score = rec.get("score")
# Validate product exists
if product_id not in valid_product_ids:
logger.warning(f"LLM returned non-existent product_id: {product_id}")
continue
# Validate score is in range
if not isinstance(score, (int, float)) or not (0 <= score <= 1):
logger.warning(f"Invalid score {score} for product {product_id}")
continue
validated.append({"product_id": product_id, "score": score})
if not validated:
logger.error("No valid recommendations after validation")
return None
return validated
except json.JSONDecodeError as e:
logger.error("Failed to parse LLM response as JSON", extra={"error": str(e)})
return None
except Exception as e:
logger.error("Unexpected error processing recommendations", extra={"error": str(e)})
return None
This validates structure, existence, and business logic constraints. When validation fails, we return a neutral result and log the issue for investigation—no crashes, no bad data leaking downstream.
No timeouts or fallbacks
LLM APIs are external services. They hang. They're slow. They fail. If your application waits indefinitely for a response, your entire request chain blocks, and customers see timeouts.
We integrated an LLM-powered help feature into a SaaS platform. The feature worked great during development. In production, during a brief API outage at the LLM provider, our support ticket creation endpoint hung for 30 seconds per request because the code was waiting for the AI summary to complete. Within seconds, the task queue backed up and the entire service became unresponsive.
Always set a timeout and always have a fallback:
import asyncio
from typing import Optional
async def get_ai_ticket_summary(ticket_text: str, timeout_seconds: int = 5) -> Optional[str]:
"""
Get AI summary with timeout and fallback.
"""
try:
response = await asyncio.wait_for(
client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=200,
messages=[{
"role": "user",
"content": f"Summarize this support ticket in one sentence: {ticket_text}"
}]
),
timeout=timeout_seconds
)
return response.content[0].text
except asyncio.TimeoutError:
logger.warning("AI summary generation timed out", extra={
"timeout_seconds": timeout_seconds,
"ticket_preview": ticket_text[:100]
})
return None
except Exception as e:
logger.error("AI summary generation failed", extra={"error": str(e)})
return None
async def create_support_ticket(text: str, user_id: str) -> dict:
"""
Create ticket with fallback if AI fails.
"""
# Try to get AI summary, but don't block ticket creation
summary = await get_ai_ticket_summary(text)
ticket = {
"user_id": user_id,
"text": text,
"summary": summary or "No AI summary available", # Fallback
"auto_categorized": summary is not None, # Track if AI was used
"created_at": datetime.utcnow()
}
await db.tickets.insert_one(ticket)
return ticket
Notice we also track whether the AI was actually used. If summaries are unavailable, we don't claim they're AI-generated. This matters for user trust and compliance.
Sending too much context
It feels safe to send everything to the LLM: the entire customer history, full database records, all related documents. More context should mean better answers, right? In practice, it creates three problems: cost, latency, and degraded quality due to noise.
We built an AI-powered search for a trading platform. For each user query, we sent the last 30 days of trades, current market data, and the user's preferences. Tokens cost money, requests took 3-5 seconds to complete, and the model was confused by irrelevant information. When we trimmed it down to only the last 5 most relevant trades and current holdings, search got faster and more accurate.
Be ruthless about context size:
def prepare_search_context(user_id: str, query: str, limit_trades: int = 5) -> str:
"""
Prepare focused context for search, not everything.
"""
# Get recent trades only
recent_trades = db.trades.find(
{"user_id": user_id}
).sort("created_at", -1).limit(limit_trades)
# Format concisely
trades_text = "\n".join([
f"- {t['symbol']} {t['side']} {t['quantity']} @ {t['price']}"
for t in recent_trades
])
# Prepare prompt
context = f"""User query: {query}
Recent trades:
{trades_text}
Respond with top 3 relevant symbols and brief reasoning."""
# Measure token count
token_count = len(context.split())
if token_count > 500: # Set a limit
logger.warning("Context exceeds token budget", extra={"tokens": token_count})
return context
In this example, we're explicit about what context goes into the prompt, limit it with limit_trades, and even warn if we exceed our token budget. This keeps requests fast, cheap, and focused.
Exposing API keys to the frontend
Never send API keys to the browser or mobile client. It's tempting—you can call the LLM directly from the frontend, reduce server load, and ship faster. The cost is total loss of control over your API usage, billing, and security.
We saw a marketplace team add a "chat with AI" feature that called OpenAI's API directly from React. Within 48 hours, a user figured out the API key was in the JavaScript bundle. They automated requests and spent $15,000 of the company's credits. More importantly, we lost all visibility into what was being asked and how our LLM quota was being used.
Always route LLM calls through your backend:
# Backend (NestJS/Express example, but logic applies to any backend)
# routes/ai.py or similar
from fastapi import APIRouter, HTTPException, Depends
from auth import require_user
router = APIRouter()
@router.post("/api/ai/chat")
async def chat(request: ChatRequest, user_id: str = Depends(require_user)):
"""
Frontend calls this endpoint, never the LLM directly.
"""
# Rate limiting per user
if await rate_limiter.is_limited(user_id, "ai_chat"):
raise HTTPException(status_code=429, detail="Too many requests")
# Validate input
if len(request.message) > 10000:
raise HTTPException(status_code=400, detail="Message too long")
# Call LLM from backend (API key never leaves here)
response = await get_llm_response(request.message)
# Validate response
if not response:
raise HTTPException(status_code=500, detail="AI service unavailable")
# Log for audit
await log_ai_usage(user_id, request.message, response)
return {"message": response}
This gives you rate limiting per user, audit logging, validation, and complete control. The frontend never sees your API key.
Common mistakes summary
| Mistake | Impact | Prevention |
|---|---|---|
| No output validation | Hallucinated data reaches users and breaks downstream systems | Parse, validate structure, check constraints before use |
| No timeout/fallback | Service hangs when LLM API is slow or down | Set explicit timeouts, provide neutral fallback, track usage |
| Excess context | Slow, expensive, and degraded quality | Send only what's necessary, measure token cost |
| Frontend API keys | Loss of control, billing abuse, security risk | Route all LLM calls through backend with auth and rate limits |
Key takeaways
- Validate every LLM output before using it. Check structure, data existence, and business logic. Gracefully degrade when validation fails.
- Set explicit timeouts on all LLM requests and provide fallbacks. External APIs are unreliable by definition.
- Minimize context sent to LLMs. More information doesn't equal better answers—focus on relevance and measure token cost.
- Never expose API keys to the frontend. Route all LLM calls through your backend where you can authenticate, rate-limit, log, and control access.
- Monitor and log LLM usage. Track latency, cost, validation failures, and timeouts. Use this data to catch problems early.
Comments
Loading comments…