Scaling Prometheus: Monitoring best practices explained

At scale, a single Prometheus instance becomes a bottleneck. You'll hit memory limits, cardinality explosions, and retention walls before you know it. Scaling Prometheus: Monitoring best practices explained requires strategy beyond just adding more disk space—it's about splitting responsibilities, choosing the right storage backend, and knowing when to shard versus when to centralize.

Why single-instance Prometheus breaks at scale

A standalone Prometheus works beautifully for small deployments: one cluster, a few hundred metrics, maybe 10–50 GB retention. But production grows. You add services, environments, and retention windows expand. Memory consumption climbs with cardinality (unique metric + label combinations), and query latency degrades.

At some point—usually when you're monitoring 500+ targets with millions of active series—you'll see Prometheus OOMKilled, scrapes taking longer than the interval, or queries timing out. That's your signal to rethink the architecture.

Vertical first — More memory and fewer series go far

Before you architect a distributed system, exhaust the single-node path. Vertical scaling is simpler, cheaper, and often solves the problem for another year.

Memory sizing

Prometheus memory usage is roughly:

• 2 KB per unique series (labels + metadata)
• Plus ingestion buffer (typically 1–2 GB)
• Plus query engine overhead

If you have 10 million active series, budget 20–25 GB just for the in-memory index. Add 5 GB for headroom and query load: you're at 30 GB minimum. A single r6i.2xlarge (61 GB RAM, 8 vCPU) can comfortably handle 50 million series.

Series reduction techniques

Before upgrading hardware, prune unnecessary metrics:

1. Drop high-cardinality labels. Don't store per-request-ID metrics. Aggregate at the source or drop them in Prometheus scrape configs.
2. Use metric relabeling. Drop metrics you don't query:
   yaml

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'app'
       static_configs:
         - targets: ['localhost:9090']
       metric_relabel_configs:
         - source_labels: [__name__]
           regex: 'go_gc_.*|process_resident_memory.*'
           action: drop

3. Increase scrape intervals. Move from 15s to 30s or 60s for non-critical metrics. This halves cardinality ingest.

Many teams cut series by 40–60% just by being ruthless about what they store. Before buying another server, audit what's actually queried in your Grafana dashboards.

Functional sharding — Separate Prometheus per team or cluster

When vertical scaling hits diminishing returns (or budget limits), split horizontally by function, not by time. Functional sharding means each Prometheus owns a logical domain: one per Kubernetes cluster, one per team, or one per geographic region.

Sharding by cluster

This is the most natural split in a multi-cluster environment:

• Cluster A (US-East): its own Prometheus, scrapes 500 targets.
• Cluster B (EU-West): separate Prometheus, scrapes 400 targets.
• A third Prometheus in your observability cluster scrapes federation endpoints from A and B for global queries.

This approach gives you blast radius isolation: an OOM in Cluster B doesn't blind you to Cluster A metrics.

Sharding by team

Alternatively, give each team (backend, platform, data) their own Prometheus:

• Backend team's Prometheus scrapes their services only.
• Platform team scrapes infrastructure and cluster-wide resources.
• Data team scrapes data pipeline services.

Each team owns their alerting rules, dashboards, and retention policy. Query isolation is natural: backend team queries their Prometheus, no cross-team contention.

Federation for aggregation

Set up a central Prometheus that scrapes federation endpoints from each shard:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'federate-cluster-a'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~".+"}'
    static_configs:
      - targets:
          - 'prometheus-a.us-east.internal:9090'

  - job_name: 'federate-cluster-b'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job=~".+"}'
    static_configs:
      - targets:
          - 'prometheus-b.eu-west.internal:9090'

The central instance pulls a subset of series (defined by the match[] params) from each shard. You keep global alerting and dashboards, while each shard stays reasonably sized.

Long-term storage — Thanos, Cortex or Mimir

Prometheus's local TSDB retention is measured in days or weeks, not months or years. For compliance, trend analysis, or post-incident forensics, you need long-term storage.

Thanos

Thanos adds a sidecar to Prometheus that uploads TSDB blocks to object storage (S3, GCS, Azure Blob). It's a thin wrapper—your Prometheus still works as before, but every 2 hours it uploads a block to S3.

• Queries span local storage (hot) and object storage (cold).
• Compaction happens asynchronously to reduce object storage cost.
• Downsampling reduces storage size for old data (1h data → 1d resolution after 30 days).

Thanos is operationally lightweight if you already run Kubernetes and S3. Drawback: querying across millions of blocks in S3 can be slow without aggressive caching.

Cortex and Mimir

Cortex (deprecated, succeeded by Mimir) is a fully distributed, multi-tenant TSDB. Instead of Prometheus scraping and Thanos uploading, services send metrics directly to Cortex via remote write. Cortex handles storage, compaction, and querying across many nodes.

• Multi-tenancy built-in: different teams' metrics isolated by tenant ID.
• Horizontally scalable: add ingesters or queriers as needed.
• Stronger consistency and query performance than Thanos for large installations.

Cortex/Mimir complexity: you're running a distributed database. More moving parts, more operational overhead. Best for teams with dedicated platform or SRE staff.

When to choose which

Remote write — Send samples to managed backends

Remote write lets Prometheus push metrics to an external system as they're scraped. No local storage, no disk I/O bottleneck—just ship raw samples to a backend.

global:
  scrape_interval: 15s

remote_write:
  - url: 'https://mimir.internal/api/prom/push'
    queue_config:
      capacity: 10000
      max_shards: 200
      max_samples_per_send: 1000
    metadata_config:
      send: true
      send_interval: 1m

This approach is popular for:

• Managed services: Grafana Cloud, Datadog, AWS Managed Prometheus. You send samples, they handle storage and querying.
• Self-hosted Mimir/Cortex: Prometheus becomes stateless. Scaling is just adding more Prometheus scrapers.
• Disaster recovery: Run two Prometheus instances, both writing to the same backend. If one dies, the other keeps shipping data.

Caveat: remote write doesn't replace local storage for queries. Prometheus can't query what it doesn't hold locally. If your remote backend goes down, you lose real-time queries. Typical production setup: keep 2–3 days local, write to remote for long-term access.

Common mistakes

• Sharding without limits. Creating 50 Prometheus instances "just in case" multiplies operational burden. Start with 2–3 functional shards, scale only when needed.
• Ignoring cardinality during growth. A single service sending http_requests{path="/api/users/123"} with per-request paths will spiral your series count. Apply relabel rules early.
• Assuming managed backends are cheaper. Per-metric pricing (Datadog) or per-GB (some Grafana Cloud tiers) can exceed self-hosted Mimir + infrastructure cost once you're at scale. Calculate before migrating.
• Federation without deduplication. Scraping the same metric from two federation sources creates duplicates. Use Thanos or deduplicate in your query layer.
• Not testing recovery. Sharded or distributed setups are invisible until a node fails. Practice losing a Prometheus instance or a Mimir ingester. Know your RTO and RPO.

Key takeaways

• Vertical scaling (bigger servers, lower cardinality) is your first line—optimize series before architecting complexity.
• Functional sharding (per cluster, per team) is simpler than technical sharding and maps cleanly to organizational ownership.
• Thanos is the lightweight play for long-term storage; Mimir/Cortex for full-blown distributed systems.
• Remote write enables stateless Prometheus and disaster recovery, but doesn't replace local storage for latency-sensitive queries.
• Choose your scaling model based on cardinality, retention window, and operational capacity—not hype.