Caching LLM responses in Practical AI & Machine Learning
Every call to a large language model costs money and time. When your team fields the same question repeatedly—whether from different users or within a single session—you're burning through tokens and watching latency climb unnecessarily. Caching LLM responses in Practical AI & Machine Learning is how we cut both cost and latency without sacrificing accuracy or freshness.
In this guide, we'll walk through four proven caching strategies: exact-match caching for deterministic lookups, semantic caching for similar questions, prompt caching to reuse expensive prefixes, and smart expiration logic to keep your cache honest when source data changes. These techniques come from real production systems handling thousands of repeated queries every day.
Exact-match cache — Hash the prompt and store the answer
The simplest and fastest cache is a direct lookup: hash the user's prompt, check if we've seen it before, and return the cached response if we have. This works brilliantly when users ask identical or near-identical questions—and in practice, this happens far more often than you'd expect in customer support, FAQ bots, and internal tools.
In production, we typically implement this with Redis or a local in-memory store, keyed by a stable hash of the prompt:
import hashlib
import json
import redis
from typing import Optional
class ExactMatchCache:
def __init__(self, redis_client: redis.Redis, ttl_seconds: int = 86400):
self.client = redis_client
self.ttl = ttl_seconds
def _prompt_hash(self, prompt: str) -> str:
"""Generate stable hash of prompt."""
return hashlib.sha256(prompt.encode()).hexdigest()
def get(self, prompt: str) -> Optional[dict]:
"""Retrieve cached response if it exists."""
key = f"llm:exact:{self._prompt_hash(prompt)}"
cached = self.client.get(key)
if cached:
return json.loads(cached)
return None
def set(self, prompt: str, response: dict) -> None:
"""Store response with TTL."""
key = f"llm:exact:{self._prompt_hash(prompt)}"
self.client.setex(key, self.ttl, json.dumps(response))
# Usage
cache = ExactMatchCache(redis.Redis())
# Check cache first
cached_response = cache.get("What is machine learning?")
if cached_response:
return cached_response
# Call LLM only on miss
response = llm_client.complete(prompt="What is machine learning?")
cache.set("What is machine learning?", response)
return responseThe key insight: exact-match caching has zero false positives. If the hash matches, the response is correct. We've run this pattern across customer support bots where 40–60% of queries are repeats, and you see immediate cost savings. The tradeoff is that a single whitespace difference or capitalization change creates a cache miss, so for user-facing applications, you may want light normalization.
Semantic cache — Reuse answers for similar questions using embeddings
Exact-match caching is efficient but narrow. A semantic cache recognizes that "What is machine learning?" and "Can you explain machine learning to me?" are asking the same thing, and reuses the answer. We do this by embedding the prompt, storing it alongside the response, and measuring cosine similarity at lookup time.
This is where caching gets interesting in production: you trade a small amount of CPU and latency on cache lookups for dramatically higher hit rates. For many teams, semantic cache hit rates jump from 20–30% to 60–75% compared to exact matching.
import numpy as np
from sentence_transformers import SentenceTransformer
import redis
import json
class SemanticCache:
def __init__(self, redis_client: redis.Redis, similarity_threshold: float = 0.85):
self.client = redis_client
self.embedder = SentenceTransformer('all-MiniLM-L6-v2')
self.threshold = similarity_threshold
def _embed_prompt(self, prompt: str) -> np.ndarray:
"""Generate embedding for prompt."""
return self.embedder.encode(prompt)
def get(self, prompt: str) -> Optional[dict]:
"""Find similar cached prompt and return response if similarity exceeds threshold."""
query_embedding = self._embed_prompt(prompt)
# Retrieve all cached embeddings (in production, use a vector DB like Pinecone or Weaviate)
keys = self.client.keys("llm:semantic:*")
best_match = None
best_similarity = 0
for key in keys:
cached_data = json.loads(self.client.get(key))
cached_embedding = np.array(cached_data['embedding'])
# Cosine similarity
similarity = np.dot(query_embedding, cached_embedding) / (
np.linalg.norm(query_embedding) * np.linalg.norm(cached_embedding)
)
if similarity > best_similarity:
best_similarity = similarity
best_match = cached_data
if best_similarity >= self.threshold:
return best_match['response']
return None
def set(self, prompt: str, response: dict) -> None:
"""Store prompt embedding and response."""
embedding = self._embed_prompt(prompt).tolist()
key = f"llm:semantic:{hashlib.sha256(prompt.encode()).hexdigest()}"
data = {
'prompt': prompt,
'embedding': embedding,
'response': response
}
self.client.setex(key, 86400, json.dumps(data))
# Usage
cache = SemanticCache(redis.Redis(), similarity_threshold=0.85)
cached_response = cache.get("Explain machine learning")
if cached_response:
return cached_responseIn practice, Redis isn't ideal for semantic search at scale; teams typically graduate to a purpose-built vector database like Pinecone, Weaviate, or Milvus. The threshold (0.85 in the example) is tunable—start conservative (0.90+) to avoid hallucination, then relax as you validate results.
Prompt caching — Many providers cache long shared prompt prefixes
Claude, GPT-4, and other modern LLM providers now offer built-in prompt caching. When you send a long system prompt or context (e.g., a 50K-token knowledge base preamble), the provider caches that prefix on their servers. Subsequent requests that include the same prefix are charged at a fraction of the normal token rate and execute faster.
This is especially valuable for multi-turn conversations where the system prompt and context remain constant, or for batch operations over the same document set.
from anthropic import Anthropic
client = Anthropic()
# Long system context that will be reused
SYSTEM_CONTEXT = """
You are a specialized medical chatbot trained on the following knowledge base:
[50,000 tokens of medical information...]
"""
def query_with_cached_prefix(user_query: str) -> str:
"""Use Claude's prompt caching for the system context."""
response = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=1024,
system=[
{
"type": "text",
"text": SYSTEM_CONTEXT,
"cache_control": {"type": "ephemeral"}
}
],
messages=[
{
"role": "user",
"content": user_query
}
]
)
# Log cache performance
usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Cache creation tokens: {usage.cache_creation_input_tokens}")
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
return response.content[0].text
# First call: cache is written
result1 = query_with_cached_prefix("What is hypertension?")
# Second call: cache is read (same prefix, different query)
result2 = query_with_cached_prefix("Explain diabetes management.")The provider tracks cache hits via usage telemetry. In our experience, prompt caching can cut costs by 30–50% when the same context is queried repeatedly, and response times improve proportionally. The catch: cache lifetimes are typically short (5 minutes for ephemeral caches, up to 24 hours for API-level caches), so this works best for concentrated query patterns.
Expire wisely — Invalidate when source data changes
A cache is only useful if it's correct. When your underlying data changes—a knowledge base is updated, a FAQ is refreshed, a document is amended—stale cached responses become liabilities. We need intelligent expiration logic.
The simplest approach is a fixed TTL (as we've shown). But production systems often need smarter invalidation:
class SmartCache:
def __init__(self, redis_client: redis.Redis):
self.client = redis_client
def set_with_dependencies(self, prompt: str, response: dict, depends_on: list[str]) -> None:
"""
Store response with explicit data dependencies.
depends_on: list of keys (e.g., knowledge base IDs) this response depends on.
"""
cache_key = f"llm:response:{hashlib.sha256(prompt.encode()).hexdigest()}"
# Store the response
self.client.setex(cache_key, 3600, json.dumps(response))
# Register this cache key as dependent on each data source
for dep in depends_on:
dep_key = f"llm:dep:{dep}"
self.client.sadd(dep_key, cache_key)
def invalidate_source(self, source_id: str) -> int:
"""Invalidate all cached responses that depend on this source."""
dep_key = f"llm:dep:{source_id}"
dependent_keys = self.client.smembers(dep_key)
deleted = 0
for key in dependent_keys:
deleted += self.client.delete(key)
self.client.delete(dep_key)
return deleted
# Usage: when a knowledge base is updated
cache = SmartCache(redis.Redis())
# Cache response with knowledge base dependency
response = llm_client.complete(prompt="What is X?")
cache.set_with_dependencies(
prompt="What is X?",
response=response,
depends_on=["kb:main", "kb:faq"]
)
# Later: knowledge base is updated
cache.invalidate_source("kb:main") # Clears all responses depending on kb:mainFor even tighter control, some teams attach version hashes to their data sources and embed them in cache keys. If the version changes, the cache key becomes invalid automatically. This is especially common in systems where knowledge bases are versioned or documents are tracked by content hash.
Common mistakes
- Caching without measuring. Always instrument cache hit rates, latency, and cost savings. Without metrics, you don't know if the complexity is worth it.
- Ignoring stale data risk. A cache with poor expiration logic is worse than no cache. Set conservative TTLs initially, then extend them as confidence grows.
- Conflating similarity with correctness. Semantic caches can cause hallucinations if the similarity threshold is too loose. Test thoroughly with domain experts before raising thresholds in production.
- Forgetting to hash sensitive data. If prompts contain user PII, never store them in plain text in the cache. Hash or tokenize before caching.
- Over-caching heterogeneous queries. If every query is unique, caching adds overhead without return. Profile first.
Key takeaways
- Exact-match caching is the safest starting point: hash the prompt, store the response, return on hit. Expect 20–40% hit rates for most real-world workloads.
- Semantic caching with embeddings can push hit rates to 60–75%, but requires a vector database and tuned similarity thresholds. Start at 0.90 similarity and relax conservatively.
- Provider-level prompt caching (Claude, GPT-4) is nearly free when available; use it for any scenario with repeated long contexts. Check usage telemetry to measure savings.
- Expiration logic must be deliberate: fixed TTLs work for many cases, but dependency tracking or version hashing is needed when source data is frequently updated.
- Instrument everything: cache hit rates, latency impact, cost savings. A cache that saves 10% on cost but adds 200ms latency is a net loss.
Comments
Loading comments…