Introduction
LLM API costs scale fast—one poorly optimized prompt loop can drain your budget in days. Reducing LLM costs in Practical AI & Machine Learning is not a nice-to-have; it's essential for any team shipping AI features to production.
We've seen teams go from $500/month to $5,000/month in a matter of weeks because they didn't think about token efficiency early. The good news is that cost reduction doesn't require cutting features or user experience. It requires discipline: caching, model selection, context trimming, and batching. This article walks through each technique with concrete examples so you can apply them to your own systems today.
Cache Repeated Prompts and Answers
The easiest win is eliminating duplicate LLM calls. If the same user asks the same question twice, or if your backend processes the same document summary five times a day, you're paying for computation you've already done.
Start with a simple in-memory cache for high-frequency queries. For a search or support platform, cache the embedding of common questions and their AI-generated answers. For document processing, cache summaries by document hash.
import hashlib
import json
from datetime import datetime, timedelta
class PromptCache:
def __init__(self, ttl_minutes=60):
self.cache = {}
self.ttl = timedelta(minutes=ttl_minutes)
def get_cache_key(self, prompt, model, temperature):
"""Generate a consistent key for a prompt + config."""
combined = f"{prompt}|{model}|{temperature}"
return hashlib.md5(combined.encode()).hexdigest()
def get(self, prompt, model, temperature):
key = self.get_cache_key(prompt, model, temperature)
if key in self.cache:
stored_at, response = self.cache[key]
if datetime.now() - stored_at < self.ttl:
return response
else:
del self.cache[key]
return None
def set(self, prompt, model, temperature, response):
key = self.get_cache_key(prompt, model, temperature)
self.cache[key] = (datetime.now(), response)
# Usage in your LLM call
cache = PromptCache(ttl_minutes=120)
def generate_summary(document_text, model="gpt-4-turbo"):
cached = cache.get(document_text, model, 0.2)
if cached:
return cached
# Call LLM only if not cached
response = openai.ChatCompletion.create(
model=model,
messages=[
{"role": "system", "content": "You are a document summarizer."},
{"role": "user", "content": f"Summarize: {document_text}"}
],
temperature=0.2
)
result = response.choices[0].message.content
cache.set(document_text, model, 0.2, result)
return resultFor production systems, graduate to Redis or a similar cache layer. Set TTLs based on how fresh your data needs to be. A product description summary might cache for 24 hours; a real-time chat message summary might cache for 10 minutes.
Also consider prompt-specific caching. If you're running the same system prompt with different user inputs, many providers now offer prompt caching features (OpenAI's cache control headers, for example). Enable these where available—they can cut costs by 50–90% on repeated calls.
Use Smaller Models for Simple Tasks
Not every task needs GPT-4. Smaller models are faster, cheaper, and often sufficient. The mistake we see most is defaulting to the largest model for everything, then wondering why costs balloon.
Profile your tasks first. Categorize them by complexity:
- Classification, extraction, simple Q&A: GPT-3.5 Turbo, Claude Haiku, Llama 2 7B
- Summarization, moderate reasoning: GPT-4 Turbo, Claude Sonnet
- Complex reasoning, multi-step logic: GPT-4, Claude 3 Opus
A routing layer lets you send requests to the right model automatically.
def route_to_model(task_type, input_length):
"""Route requests to cost-effective models."""
# Short, simple extraction tasks
if task_type == "extract_fields" and input_length < 500:
return "gpt-3.5-turbo"
# Classification
if task_type == "classify":
return "gpt-3.5-turbo"
# Summarization of moderate length
if task_type == "summarize" and input_length < 2000:
return "gpt-4-turbo"
# Long document summarization or complex reasoning
if task_type == "summarize" and input_length >= 2000:
return "gpt-4"
if task_type == "reasoning":
return "gpt-4"
# Default fallback
return "gpt-4"
def process_request(task, payload):
model = route_to_model(task["type"], len(payload.get("text", "")))
return call_llm(model, task["prompt"], payload)We've seen teams cut costs by 40% just by moving classification and extraction off their largest model. The tradeoff is occasionally lower quality, so A/B test with real users before rolling out to 100%.
Trim Context to What Is Needed
LLMs charge by tokens, and every token in your context window costs money. Long system prompts, irrelevant document excerpts, and verbose chat histories add up fast. Trimming context is one of the highest-ROI optimizations you can make.
Start by auditing what context you're actually sending. Log the token count before each API call and analyze your distribution. You might find that 80% of your calls include 10x more context than necessary.
import tiktoken
def count_tokens(text, model="gpt-4"):
"""Count tokens for a given text."""
encoder = tiktoken.encoding_for_model(model)
return len(encoder.encode(text))
def trim_context(documents, max_tokens=1000, model="gpt-4"):
"""Trim documents to fit a token budget."""
encoder = tiktoken.encoding_for_model(model)
trimmed = []
token_count = 0
for doc in documents:
doc_tokens = len(encoder.encode(doc))
if token_count + doc_tokens <= max_tokens:
trimmed.append(doc)
token_count += doc_tokens
else:
# Truncate last document to fit
remaining = max_tokens - token_count
if remaining > 0:
truncated = encoder.decode(
encoder.encode(doc)[:remaining]
)
trimmed.append(truncated)
break
return trimmed, token_count
# Example: retrieve relevant docs but only include top 3
def answer_question(question, all_documents):
# Rank by relevance (use embedding similarity, BM25, etc.)
top_docs = retrieve_top_k(question, all_documents, k=3)
# Trim to token budget
trimmed_docs, tokens_used = trim_context(top_docs, max_tokens=1500)
context = "\n\n".join(trimmed_docs)
response = call_llm(
model="gpt-4",
system="You are a helpful assistant.",
user_prompt=f"Context:\n{context}\n\nQuestion: {question}"
)
return responseAlso audit your system prompts. A 500-token system prompt repeated across 10,000 daily requests is 5 million tokens—about $75 at GPT-4 Turbo rates. Compress it: remove examples you don't need, use bullet points instead of prose, reference external docs instead of embedding them in the prompt.
For chat applications, implement a sliding window. Keep only the last 5–10 exchanges, not the entire conversation history. Summarize older messages into a brief recap if context is critical.
Batch Offline Work
Real-time API calls are convenient but expensive. Batch processing—where you queue work and process it in bulk—can reduce per-request costs by 30–50% and give you better throughput.
If you're generating summaries, descriptions, or tags for a large dataset, don't process one item at a time in your web request handler. Queue them, then process in batches during off-peak hours or in a background job.
import json
from datetime import datetime
from queue import Queue
class BatchLLMProcessor:
def __init__(self, batch_size=50, max_wait_minutes=60):
self.queue = []
self.batch_size = batch_size
self.max_wait_minutes = max_wait_minutes
self.earliest_queued = None
def enqueue(self, task_id, prompt, model="gpt-4-turbo"):
"""Add a task to the batch queue."""
self.queue.append({
"task_id": task_id,
"prompt": prompt,
"model": model,
"queued_at": datetime.now()
})
if self.earliest_queued is None:
self.earliest_queued = datetime.now()
# Trigger batch if we hit size threshold or time threshold
if len(self.queue) >= self.batch_size:
return self.process_batch()
if (datetime.now() - self.earliest_queued).total_seconds() > self.max_wait_minutes * 60:
return self.process_batch()
return None
def process_batch(self):
"""Process queued tasks in one batch call."""
if not self.queue:
return []
tasks = self.queue
self.queue = []
self.earliest_queued = None
# Call LLM with all prompts
results = []
for task in tasks:
response = call_llm(
model=task["model"],
prompt=task["prompt"]
)
results.append({
"task_id": task["task_id"],
"response": response
})
return results
# Usage: queue requests from your web app
processor = BatchLLMProcessor(batch_size=100)
def enqueue_product_summary(product_id, product_text):
"""Queue a summary job instead of calling API immediately."""
processor.enqueue(
task_id=f"summary_{product_id}",
prompt=f"Summarize this product: {product_text}",
model="gpt-3.5-turbo"
)
# Return immediately; summary will be ready in a few seconds to minutes
# Run this on a scheduled job (e.g., every 5 minutes)
def process_pending_llm_work():
results = processor.process_batch()
for result in results:
store_result(result["task_id"], result["response"])Batching is particularly effective for:
- Daily or hourly bulk operations (tag generation, categorization)
- Non-urgent user features (sending a summary email later, not immediately)
- Data processing pipelines (ETL, knowledge base ingestion)
The trade-off is latency. You can't do this for synchronous user requests, but you can for background jobs, scheduled exports, or async notifications.
Common Mistakes
Mistake 1: Not monitoring costs. Set up cost alerts in your LLM provider dashboard. If your daily spend spikes 2x, you want to know immediately, not at month-end.
Mistake 2: Premature optimization. Don't cache everything or use tiny models if quality suffers. Start with the highest-cost flows, profile them, then optimize.
Mistake 3: Ignoring context waste. We often see teams sending 5,000 tokens of context when 500 would do. Audit your actual usage before tuning.
Mistake 4: Not testing model downgrades. Moving from GPT-4 to GPT-3.5 Turbo for a task is a 10x cost reduction, but only if quality is acceptable. Set up acceptance tests before rolling out.
Mistake 5: Fire-and-forget batching. If your batch job crashes, all queued work is lost. Persist the queue to a database (Redis, Postgres) so you can retry.
Key Takeaways
- Cache repeated prompts and responses using Redis or in-memory stores, with TTLs based on your data freshness requirements.
- Route tasks to smaller models (GPT-3.5, Haiku) for classification, extraction, and simple Q&A; reserve large models for complex reasoning.
- Trim context aggressively: audit token counts, compress system prompts, and use sliding windows for chat histories.
- Batch non-urgent work offline to reduce per-request costs and improve throughput during off-peak hours.
- Monitor costs continuously and profile before optimizing to avoid wasting effort on low-impact changes.
Comments
Loading comments…