Monitoring Python apps: a practical guide for developers

Monitoring Python apps: a practical guide for developers

Most of us deploy Python services to production and hope the logs tell us when things go wrong. By the time you're woken up at 3 AM by a customer complaint, you've already lost visibility into what happened in the hours before the failure. In this guide, we'll walk through monitoring Python apps with Prometheus and Grafana, covering the practical tooling and patterns that work in FastAPI and Django services handling real traffic.

Why Python monitoring differs from other stacks

Python's single-threaded execution model (thanks to the GIL) means your production workload typically runs across multiple processes. This affects how you export metrics. A Node.js app can push metrics from a single process; your gunicorn setup with eight workers needs a different approach. We'll cover that challenge head-on because it catches most teams off guard.

The prometheus_client library

The foundation for any Prometheus integration in Python is the prometheus_client library. It provides four core metric types: Counter, Gauge, Histogram, and Summary. Think of Counter for request counts that only go up, Gauge for values that rise and fall (like active connections), Histogram for latency buckets, and Summary for percentile calculations.

Here's a minimal example that instruments a simple endpoint:

from prometheus_client import Counter, Gauge, Histogram, start_http_server
import time

# Define metrics
request_count = Counter('http_requests_total', 'Total HTTP requests', ['method', 'endpoint'])
request_duration = Histogram('http_request_duration_seconds', 'HTTP request latency', buckets=(0.01, 0.05, 0.1, 0.5, 1.0))
active_connections = Gauge('http_active_connections', 'Number of active connections')

# Start metrics server on port 8001
start_http_server(8001)

def handle_request():
    with request_duration.time():
        active_connections.inc()
        try:
            time.sleep(0.1)  # Simulate work
            request_count.labels(method='GET', endpoint='/users').inc()
        finally:
            active_connections.dec()

if __name__ == '__main__':
    for _ in range(10):
        handle_request()

The metrics endpoint becomes available at http://localhost:8001/metrics. Each metric has a name, description, labels for dimensionality, and a value. Labels like method and endpoint let you slice metrics by request type in Grafana without needing separate metric definitions.

One thing to watch: every label combination creates a new time series. If you label by user ID or request ID, you'll explode cardinality and destroy Prometheus performance. Stick to low-cardinality labels like HTTP method, status code, or service region.

prometheus-fastapi-instrumentator for automatic metrics

Manually wrapping every endpoint gets tedious fast. For FastAPI services, prometheus-fastapi-instrumentator automatically collects request latency, status codes, and method distributions across all routes.

Setup takes three lines:

from fastapi import FastAPI
from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

Instrumentator().instrument(app).expose(app)

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    return {"user_id": user_id, "name": "Alice"}

The instrumentator middleware now intercepts every request and records it. Visit http://localhost:8000/metrics to see the output. You'll get metrics like http_requests_total, http_request_duration_seconds, and status code breakdowns automatically.

If you need custom metrics inside a route, you can inject them:

from prometheus_client import Counter
from fastapi import FastAPI, Request

app = FastAPI()
Instrumentator().instrument(app).expose(app)

db_errors = Counter('db_errors_total', 'Database errors', ['operation'])

@app.get("/users/{user_id}")
async def get_user(user_id: int):
    try:
        # Query database
        user = fetch_from_db(user_id)
    except DatabaseError:
        db_errors.labels(operation='fetch_user').inc()
        raise
    return user

For Django, you'll reach for django-prometheus, which hooks into middleware and the ORM to automatically record request metrics and database query times. The setup is similarly straightforward: add it to INSTALLED_APPS and MIDDLEWARE, and you're collecting data.

Multiprocess mode for Gunicorn

Here's where most Python monitoring setups stumble. When you run gunicorn with eight workers, each process has its own in-memory metric registry. A Counter incremented in worker 1 doesn't exist in worker 2. If a client scrapes metrics and hits worker 1 one time and worker 3 the next, the totals look inconsistent.

Prometheus client solves this with multiprocess mode. Instead of storing metrics in process memory, they're written to temporary files on disk. When Prometheus scrapes the metrics endpoint, a collector reads all worker files and aggregates them.

First, install the multiprocess helper:

pip install prometheus-client[twisted]

Then set an environment variable before starting gunicorn:

export PROMETHEUS_MULTIPROC_DIR=/tmp/prometheus_multiproc
mkdir -p /tmp/prometheus_multiproc
gunicorn -w 8 -b 0.0.0.0:8000 app:app

In your FastAPI or Django app, detect multiprocess mode and configure accordingly:

import os
from prometheus_client import CollectorRegistry, generate_latest, REGISTRY
from prometheus_client.multiprocess import MultiProcessCollector

# In your app startup or conftest
if "PROMETHEUS_MULTIPROC_DIR" in os.environ:
    registry = CollectorRegistry()
    MultiProcessCollector(registry)
else:
    registry = REGISTRY

Now metrics files accumulate in your temp directory. When Prometheus scrapes, it reads all of them and merges the values. This approach works seamlessly with orchestrated deployments. If you're running in Kubernetes with a sidecar or hostPath volume mounted to that temp directory, Prometheus can reach the aggregated metrics reliably.

One caveat: remember to clean up old temp files periodically. A cron job or init container that removes files older than an hour prevents disk space issues.

Histogram request durations and understanding buckets

Histograms are where latency monitoring gets powerful. Instead of storing every request duration, Prometheus stores counts in predefined buckets. A request taking 87ms lands in the 0.1s bucket. You can calculate percentiles afterward in Grafana.

The default buckets often don't match your service. A microsecond-sensitive trading engine needs buckets=(0.001, 0.01, 0.1, 1.0). An image resizing service might use buckets=(0.5, 1.0, 2.0, 5.0, 10.0).

Here's how to define thoughtful buckets for a typical API:

from prometheus_client import Histogram

# Buckets in seconds: 10ms, 50ms, 100ms, 500ms, 1s, 5s
http_request_duration = Histogram(
    'http_request_duration_seconds',
    'Request latency',
    ['method', 'endpoint'],
    buckets=(0.01, 0.05, 0.1, 0.5, 1.0, 5.0)
)

@app.get("/expensive-operation")
async def expensive():
    with http_request_duration.labels(method='GET', endpoint='/expensive-operation').time():
        # Your logic here
        result = await do_heavy_work()
    return result

In Grafana, you query http_request_duration_seconds_bucket to see counts per bucket. The query histogram_quantile(0.95, http_request_duration_seconds_bucket) gives you the 95th percentile latency. The _sum and _count` suffixes let you calculate average latency across your scrape interval.

A common mistake is setting too many buckets. Each bucket creates a separate time series. Twenty buckets across five label combinations means 100 time series just for that one metric. Keep it lean: seven to ten buckets cover most use cases.

Common mistakes

• Forgetting multiprocess mode in production. Your metrics are correct locally but inconsistent under load because workers are writing to separate registries. Always test with multiple gunicorn workers locally before deploying.
• High-cardinality labels. Labeling by request ID, user ID, or request path as a dimension creates a time series explosion. Prometheus scrapes will get slower, and cardinality limits will bite you. Use low-cardinality dimensions like method and status code.
• Starting metrics server on all interfaces without authentication. Your /metrics endpoint exposes internal service details. Run it on localhost or behind a firewall, or require authentication in production.
• Not tuning histogram buckets. Default buckets are optimized for generic web services. Services with different latency profiles (batch jobs, real-time systems) need custom buckets or they hide important percentiles.
• Ignoring metric naming conventions. Prometheus prefers request_duration_seconds over req_dur_ms. Unit suffixes matter for alerting and cross-service consistency. Follow the standard naming scheme.

Key takeaways

• Use prometheus_client for custom metrics and prometheus-fastapi-instrumentator or django-prometheus for automatic HTTP instrumentation.
• Enable multiprocess mode with PROMETHEUS_MULTIPROC_DIR when running multiple gunicorn workers to ensure metric consistency.
• Define histograms with buckets matched to your service's latency profile, not generic defaults.
• Keep labels low-cardinality and follow Prometheus naming conventions to avoid performance cliffs.
• Expose metrics on localhost or behind authentication in production; never leave /metrics public.