Calling LLM APIs from a backend: lessons from production

Calling LLM APIs from a backend: lessons from production

When you add AI features to your product, the first instinct is often to call an LLM API directly from your frontend. That's a mistake we've seen cost teams months of debugging and security incidents. Calling LLM APIs from a backend is the only pattern that works at scale, and it comes with its own set of hard-won lessons from teams running these systems in production.

Over the last six months, we've built AI-powered search, chat interfaces, and content generation across multiple SaaS products. Each time, the same problems emerged—and each time, the solutions were the same. This article captures what we've learned about keeping credentials safe, making responses feel instant, handling failures gracefully, and observing what your LLM is actually doing in production.

Keep keys server-side — Never call LLM APIs directly from the browser

This is the first rule, and it's non-negotiable. If your frontend has access to your OpenAI, Anthropic, or other LLM provider API key, you've already lost.

When keys live in your browser, three things happen. First, they're visible in your network tab, your browser cache, and every malicious script that runs on your page. Second, your rate limits and billing are exposed to any user who knows how to read DevTools. Third, you have no way to control who can call your LLM, or to audit what they're asking it to do.

The solution is straightforward: your backend owns the LLM credentials. Your frontend calls your backend. Your backend calls the LLM provider. Your frontend never sees the key.

Here's what this looks like in practice. Your Next.js or React app sends a request to your backend with the user's input:

// Frontend (Next.js/React) - safe to expose
async function generateAnswer(question) {
  const response = await fetch('/api/ai/answer', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ question })
  });
  return response.json();
}

Your backend receives this, validates it, and calls the LLM with credentials it controls:

// Backend (Node.js/Python) - credentials stay here
from openai import OpenAI

client = OpenAI(api_key=os.environ['OPENAI_API_KEY'])

def generate_answer(question: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": question}]
    )
    return response.choices[0].message.content

Your backend also becomes the place where you validate input (is this user allowed to ask this question?), rate-limit (has this user burned through their quota?), and audit (what questions are being asked?). You can't do any of that if the key lives in the browser.

Stream responses — Server-sent events make answers feel instant

LLM API calls are slow. GPT-4 can take 10–30 seconds to respond to a complex prompt. If your user has to wait for the entire response to arrive before they see anything on screen, they'll think your app is broken.

Streaming—where the LLM sends back tokens one at a time—solves this. Your user sees the answer start appearing after 500ms, and it feels responsive even though the full response takes 20 seconds.

You have two options: WebSockets or Server-Sent Events (SSE). For most teams, SSE is simpler. It's built into the HTTP spec, requires no separate connection management, and works through most proxies and CDNs.

Here's the pattern. Your frontend initiates a request and listens for streamed events:

// Frontend - listen to streamed response
function streamAnswer(question) {
  const eventSource = new EventSource(
    `/api/ai/answer-stream?question=${encodeURIComponent(question)}`
  );
  
  eventSource.onmessage = (event) => {
    const token = JSON.parse(event.data).token;
    appendToScreen(token); // user sees text appear in real-time
  };
  
  eventSource.onerror = () => {
    eventSource.close();
  };
}

Your backend streams tokens as they arrive from the LLM provider:

from flask import Flask, Response
import json

@app.route('/api/ai/answer-stream')
def answer_stream():
    question = request.args.get('question')
    
    def generate():
        # Most LLM SDKs support streaming
        with client.chat.completions.create(
            model="gpt-4",
            messages=[{"role": "user", "content": question}],
            stream=True  # Key: enable streaming
        ) as response:
            for chunk in response:
                token = chunk.choices[0].delta.content
                if token:
                    yield f"data: {json.dumps({'token': token})}\n\n"
    
    return Response(
        generate(),
        mimetype='text/event-stream',
        headers={
            'Cache-Control': 'no-cache',
            'Connection': 'keep-alive',
            'X-Accel-Buffering': 'no'  # Important for proxies
        }
    )

The X-Accel-Buffering: no header is critical if you're behind Nginx or similar proxies. Without it, they buffer the response and you don't get the streaming benefit.

Timeouts and retries — LLM calls are slow and sometimes fail

LLM APIs fail. Not often, but often enough that you can't ignore it. Rate limits, overloaded endpoints, and transient network issues mean your call will occasionally time out or return a 429 (too many requests) error.

Set a timeout on every LLM call. If it takes longer than, say, 60 seconds, stop waiting and tell the user something went wrong. Then retry with exponential backoff—wait 1 second, then 2 seconds, then 4 seconds, up to a maximum number of attempts.

Here's what this looks like:

import time
from openai import RateLimitError, APITimeoutError

def call_llm_with_retries(question: str, max_retries: int = 3) -> str:
    for attempt in range(max_retries):
        try:
            response = client.chat.completions.create(
                model="gpt-4",
                messages=[{"role": "user", "content": question}],
                timeout=60  # Don't wait forever
            )
            return response.choices[0].message.content
        except (RateLimitError, APITimeoutError) as e:
            if attempt == max_retries - 1:
                raise  # Last attempt, give up
            wait_time = 2 ** attempt  # 1, 2, 4 seconds
            time.sleep(wait_time)
        except Exception as e:
            # Other errors: auth failure, bad prompt, etc.
            # Don't retry these
            raise

In your frontend, also set a timeout. If the server hasn't started streaming tokens after 30 seconds, disconnect and tell the user to try again:

// Frontend timeout
function streamAnswer(question) {
  const timeoutId = setTimeout(() => {
    eventSource.close();
    showError('Request took too long. Please try again.');
  }, 30000);
  
  const eventSource = new EventSource(`/api/ai/answer-stream?question=...`);
  
  eventSource.onmessage = () => clearTimeout(timeoutId);
  eventSource.onerror = () => clearTimeout(timeoutId);
}

One subtlety: be careful with streaming responses and retries. Once you start streaming to the client, you can't retry the whole call—the user has already seen part of the response. Either commit to the response and show an error at the end if something goes wrong, or add a confirmation step before you start streaming.

Log prompts and outputs — With care for personal data — needed for debugging and evaluation

You need to log what's going into your LLM and what's coming back out. Without logs, you can't debug why a response was wrong, you can't evaluate your prompts, and you can't detect misuse. But logging user data requires care.

Here's the pattern we use. Log every request and response, but be explicit about what you're storing and who can see it:

import logging
import json
from datetime import datetime

logger = logging.getLogger(__name__)

def log_llm_call(user_id: str, question: str, response: str, model: str):
    log_entry = {
        'timestamp': datetime.utcnow().isoformat(),
        'user_id': user_id,
        'model': model,
        'question': question[:500],  # Truncate very long inputs
        'response': response[:1000],  # Truncate very long outputs
        'input_tokens': len(question.split()),
        'output_tokens': len(response.split())
    }
    logger.info(json.dumps(log_entry))

Store these logs in a place only your data team can access—not in your general application logs. If a user asks to delete their data, you must delete their corresponding log entries too.

Better yet, use a sampling approach. Log 10% of requests in detail, but log metadata (model, latency, token count, success/failure) for all of them. This gives you signal without storing massive amounts of user data:

import random

def maybe_log_llm_call(user_id: str, question: str, response: str, model: str):
    # Always log metadata
    metrics_logger.info({
        'event': 'llm_call',
        'user_id': user_id,
        'model': model,
        'latency_ms': elapsed_time,
        'success': True,
        'input_length': len(question),
        'output_length': len(response)
    })
    
    # Sample detailed logs
    if random.random() < 0.1:
        detailed_logger.info({
            'question': question,
            'response': response
        })

With this setup, you can answer questions like "what was the latency of GPT-4 calls yesterday?" for all traffic, but only have detailed question/response pairs for 10% of users. When you need to debug a specific issue, you can temporarily increase the sampling rate or pull logs for a specific user with their consent.

Common mistakes

• Calling the LLM synchronously in the request handler. This blocks your entire backend while waiting for the response. Use a job queue (Celery, Bull, Temporal) to call the LLM asynchronously and fetch the result when the frontend asks for it, or stream back directly as shown above.
• Forgetting to validate user input before sending it to the LLM. A malicious user can craft prompts that cause the LLM to ignore instructions or expose system details. Always validate, truncate, and sanitize inputs on the backend before passing them to the API.
• Not handling streaming disconnects. If a user closes their browser tab mid-stream, your backend keeps processing and sending tokens to no one. Use context managers and ensure your streaming handler cleans up gracefully.
• Mixing secrets in logs. Never log the full prompt if it contains user passwords, API keys, or other sensitive data. Redact these before logging.
• Setting timeouts too short. A 5-second timeout on an LLM call is too aggressive. Most calls take 10–30 seconds. Aim for 60 seconds and adjust based on your traffic and provider SLAs.

Key takeaways

• Keep LLM API keys in environment variables on your backend. Never expose them to the frontend.
• Use Server-Sent Events to stream LLM responses token-by-token. This makes responses feel instant even though the full call takes 10–30 seconds.
• Set timeouts (60 seconds) and retry with exponential backoff (1, 2, 4 seconds). LLM APIs are slower and less reliable than typical web services.
• Log prompts, responses, and metadata, but be careful about user privacy. Use sampling, truncation, and separate storage for detailed logs.
• Validate and sanitize all user input on the backend before passing it to the LLM provider.
• Always include X-Accel-Buffering: no in streaming responses if you're behind a proxy.