Caching in Django: a Python Backend guide for developers

Caching in Django: a Python Backend guide for developers

A typical e-commerce dashboard query that joins users, orders, and payment records can take 2–3 seconds on a cold database. Run that same page 100 times a day and your database is doing redundant work that could be eliminated with caching. Caching in Django: a Python Backend guide for developers covers the strategies we use in production to speed up both page rendering and expensive computations, from simple per-view caching to Redis-backed distributed caches that scale across multiple servers.

Why caching matters in Django applications

Before we dive into implementation, let's be clear about what caching solves. Database queries, API calls, and CPU-intensive calculations are the three main bottlenecks in backend systems. A user-facing dashboard might spend 60% of its response time fetching and aggregating data that hasn't changed in the last five minutes. A background job computing fraud scores might recalculate the same thresholds every time it runs. Without caching, your database connection pool fills up, response times spike, and operational costs climb.

Django gives us three levels of caching strategy: per-view (fastest for whole pages), low-level cache API (granular, flexible), and backend selection (where you store the cache). We'll walk through each, then show how to keep cached data fresh.

Per-view caching with cache_page

The simplest caching strategy is to cache the entire rendered HTTP response. Django's cache_page decorator does exactly that: it stores the response and serves the same bytes to every user until the cache expires. This works well for public pages that don't depend on user identity or session state.

Imagine a product listing page that shows inventory across 50,000 SKUs. The query runs in 1.5 seconds, and we get 500 requests per minute during peak hours. Without caching, that's 750 seconds of database time per minute. With a 60-second cache, we reduce it to 1.5 seconds:

from django.views.decorators.cache import cache_page
from django.http import JsonResponse
from products.models import Product

@cache_page(60)
def product_list(request):
    products = Product.objects.values('id', 'name', 'price').all()[:50000]
    return JsonResponse({
        'products': list(products),
        'count': len(products)
    })

The decorator intercepts the request before your view runs. If a matching cache key exists (based on URL and request method), the cached response is returned immediately. If not, your view executes, and Django caches the response automatically.

When to use cache_page: Public pages, list views, feed endpoints, anything that's identical for all users. Do not use it on pages with user-specific content, CSRF tokens, or dynamically generated session data.

Cache key generation: By default, Django uses the full request path and query string to build the cache key. For GET /products/?category=electronics, the key includes the query parameters. POST requests are never cached by cache_page.

Low-level cache API for granular control

Not everything fits the cache_page model. You might need to cache part of a response, or cache data that spans multiple views. The low-level cache API gives you explicit control.

The most common pattern is cache.get_or_set: try to fetch from cache, and if it misses, compute the value and store it. Here's a real scenario from a SaaS analytics dashboard:

from django.core.cache import cache
from django.http import JsonResponse
from analytics.models import Event

def dashboard_stats(request):
    # Expensive computation: aggregates 10M+ events
    def compute_stats():
        return {
            'total_revenue': Event.objects.filter(
                type='purchase'
            ).aggregate(total=Sum('amount'))['total'],
            'active_users_today': Event.objects.filter(
                created_at__gte=timezone.now() - timedelta(hours=24)
            ).values('user_id').distinct().count(),
            'top_events': list(
                Event.objects.values('type')
                .annotate(count=Count('id'))
                .order_by('-count')[:10]
            )
        }
    
    # Cache for 5 minutes (300 seconds)
    stats = cache.get_or_set('dashboard_stats', compute_stats, 300)
    return JsonResponse(stats)

On the first request, compute_stats() runs and the result is stored in cache. On the next 59 requests within 5 minutes, the cached value is returned instantly. After 300 seconds, the cache expires and the next request triggers a fresh computation.

Other common cache operations:

• cache.get(key) — Fetch a value, return None if it doesn't exist.
• cache.set(key, value, timeout) — Store a value explicitly.
• cache.delete(key) — Remove a value (critical for invalidation).
• cache.delete_many(keys) — Remove multiple keys in one operation.
• cache.get_many(keys) — Fetch multiple keys, useful for batching.

For a real-world example, here's how we cache expensive permission checks in a ride-hailing service:

from django.core.cache import cache

def can_user_access_ride(user_id, ride_id):
    cache_key = f'user_access:{user_id}:{ride_id}'
    
    # First check: is this permission already cached?
    has_access = cache.get(cache_key)
    if has_access is not None:
        return has_access
    
    # Miss: compute the permission from the database
    try:
        ride = Ride.objects.get(id=ride_id)
        has_access = (
            user_id == ride.driver_id or 
            user_id == ride.passenger_id or
            User.objects.get(id=user_id).is_support_staff
        )
    except Ride.DoesNotExist:
        has_access = False
    
    # Cache for 10 minutes
    cache.set(cache_key, has_access, 600)
    return has_access

This pattern eliminates repeated database lookups for the same permission query and scales well because the cache key is specific to the user and resource.

Redis as the production cache backend

Django ships with an in-memory cache backend (Memcached or a local memory cache), but in production we need something that persists across server restarts, supports expiration, and scales to multiple application servers. Redis is the industry standard.

Configure it in settings.py:

CACHES = {
    'default': {
        'BACKEND': 'django_redis.cache.RedisCache',
        'LOCATION': 'redis://127.0.0.1:6379/1',
        'OPTIONS': {
            'CLIENT_CLASS': 'django_redis.client.DefaultClient',
            'SOCKET_CONNECT_TIMEOUT': 5,
            'SOCKET_TIMEOUT': 5,
            'CONNECTION_POOL_KWARGS': {'max_connections': 50},
        }
    }
}

A few key settings:

• LOCATION: Redis connection string. Use redis:// for standard TCP or rediss:// for TLS.
• max_connections: Connection pool size. Set based on your application's concurrency (typically 10–50).
• SOCKET_TIMEOUT: How long to wait for Redis to respond. If Redis is slow, set this low to fail fast rather than block your threads.

In AWS or managed hosting, use a managed Redis service (ElastiCache, DigitalOcean, etc.). For self-hosted, run Redis on a dedicated instance or container with persistence enabled (RDB snapshots or AOF).

Why Redis over Memcached? Redis supports data types (sets, sorted sets, hashes) beyond strings, which enables more sophisticated caching patterns. It also persists data, supports pub/sub for invalidation, and has better observability. For most Django teams, Redis is the right choice.

Cache invalidation: keeping data fresh

Caching solves the read problem but introduces a consistency problem: if you cache a user's profile and then the user updates their name, the cache still serves the old name. Cache invalidation is harder than caching itself, and the classic joke is that there are only two hard things in computer science: cache invalidation and naming things.

The strategy is simple: delete the cache key whenever the underlying data changes. In Django, use signals to automate this:

from django.db.models.signals import post_save, post_delete
from django.dispatch import receiver
from django.core.cache import cache
from users.models import User

@receiver(post_save, sender=User)
def invalidate_user_cache(sender, instance, **kwargs):
    """Clear cached data whenever a user is saved."""
    cache.delete(f'user_profile:{instance.id}')
    cache.delete(f'user_access:{instance.id}:*')  # Pattern delete
    cache.delete(f'dashboard_stats')

@receiver(post_delete, sender=User)
def invalidate_user_on_delete(sender, instance, **kwargs):
    """Clear cached data when a user is deleted."""
    cache.delete(f'user_profile:{instance.id}')

Now whenever a User is saved or deleted, the associated cache keys are automatically cleared. The next request will recompute and cache the fresh data.

Pattern-based invalidation: Some cache backends (like Redis) support wildcard deletions. Be cautious: cache.delete_pattern('user_access:*') might be expensive if you have millions of keys. Instead, be specific: cache.delete_many([f'user_access:{user_id}:{rid}' for rid in ride_ids]).

For distributed systems, consider using cache tags or a separate cache layer. Some teams use a version number in the key:

def get_user_profile(user_id):
    version = cache.get(f'user_version:{user_id}', 0)
    cache_key = f'user_profile:{user_id}:v{version}'
    profile = cache.get(cache_key)
    if not profile:
        profile = User.objects.get(id=user_id).to_dict()
        cache.set(cache_key, profile, 3600)
    return profile

def invalidate_user_profile(user_id):
    # Increment version number; old cache keys become unreferenced
    cache.incr(f'user_version:{user_id}', delta=1)

This avoids deletes entirely: you just bump a version number, and old cache keys expire naturally.

Common mistakes

• Caching user-specific data with cache_page: If your page has personalization, session tokens, or CSRF tokens, @cache_page will serve the same response to all users. Use the low-level API instead.
• Forgetting to invalidate on writes: The most common source of bugs. Always delete or expire cache keys when the underlying data changes. Use signals to automate this.
• Setting cache timeout too high: A 24-hour cache is safer than a 5-minute cache if you forget invalidation, but it's also wrong for most use cases. For data that changes frequently, use 5–30 minutes. For reference data (product catalogs, feature flags), use 1–4 hours.
• Not monitoring cache hit rate: If your cache hit rate is below 50%, you're not caching effectively. Use Redis monitoring tools (redis-cli INFO stats, DataDog, New Relic) to track hits and misses.
• Caching None values: If a query returns no results, you can cache None, but set a shorter timeout (60–120 seconds) to avoid serving stale "not found" responses.
• Serialization errors: Cache values must be picklable. Complex Django model instances, file handles, and database connections cannot be cached. Convert to dictionaries or use custom serializers.

Key takeaways

• Use @cache_page for public, static pages; use the low-level cache API for granular, data-dependent caching.
• cache.get_or_set() is your workhorse: it eliminates the get-compute-set boilerplate and handles cache misses gracefully.
• Deploy Redis in production, not the memory cache. Redis is persistent, scalable, and provides better observability.
• Invalidate cache keys when data changes using Django signals. Automate this; don't rely on manual cache clears.
• Monitor cache hit rates and watch for stale data. A cache that's wrong is worse than no cache at all.
• Set reasonable timeouts: 5–30 minutes for frequently changing data, 1–4 hours for reference data. Shorter timeouts reduce staleness risk; longer timeouts reduce database load.