Thanos and Mimir: Monitoring best practices explained

By day 51 of our monitoring journey, you've likely hit the wall with a single Prometheus instance—it's running out of storage, queries are slowing down, and you can't see metrics across your infrastructure. Thanos and Mimir: Monitoring best practices explained addresses exactly this pain point by showing you how to scale Prometheus horizontally, retain metrics indefinitely, and query your entire fleet as one coherent system. Both projects take different architectural approaches to solve the same problem: turning Prometheus from a single-node scraper into an enterprise-grade monitoring platform.

Store blocks in object storage

The core innovation behind both Thanos and Mimir is treating Prometheus as a time-series factory rather than a database. Prometheus still scrapes and writes data locally, but instead of keeping everything in its embedded TSDB forever, it regularly compacts metrics into immutable blocks and uploads them to object storage—S3, GCS, or MinIO.

When you run Prometheus with Thanos Sidecar, you're adding a companion process that watches your Prometheus data directory. Every two hours (by default), Prometheus compacts recent metrics into a block directory. Thanos detects these blocks, uploads them to your object storage backend, and then can safely delete local files to keep disk usage bounded. This is crucial: you're no longer limited by a single node's disk capacity.

global:
  scrape_interval: 15s
  external_labels:
    cluster: eu-prod
    replica: 0

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'kubernetes'
    kubernetes_sd_configs:
      - role: pod

That's your basic Prometheus config. Now add the Thanos sidecar as a separate container in your pod:

- name: thanos-sidecar
  image: quay.io/thanos/thanos:v0.35.0
  args:
    - sidecar
    - --tsdb.path=/prometheus
    - --prometheus.url=http://localhost:9090
    - --objstore.config-file=/etc/thanos/objstore.yaml
  volumeMounts:
    - name: prometheus-storage
      mountPath: /prometheus
    - name: thanos-objstore-config
      mountPath: /etc/thanos

The object store config tells Thanos where to upload blocks. For S3-compatible storage:

type: s3
config:
  bucket: my-prometheus-blocks
  endpoint: s3.us-east-1.amazonaws.com
  access_key: ${AWS_ACCESS_KEY_ID}
  secret_key: ${AWS_SECRET_ACCESS_KEY}

Mimir takes a more opinionated approach: it replaces Prometheus entirely. You don't run Prometheus; instead, you point your scrape jobs directly at Mimir's remote write endpoint. Mimir handles all storage internally, using object storage as its backing store. This simplifies the architecture but means you're committing to Mimir's operational model rather than extending an existing Prometheus setup.

The practical benefit is enormous. Instead of provisioning 500GB disks for production Prometheus instances, you provision 50GB local disks and let object storage absorb unlimited history. Your infrastructure cost shifts from expensive local SSD to cheaper S3.

Global query view across clusters

Once you have blocks in object storage, the next piece is the query layer. You might have Prometheus scraping in your US region, EU region, and development cluster. Thanos Query (or Mimir's query frontend) acts as a unified gateway that can answer questions like "what's the CPU usage across all clusters?" without you having to run three separate Prometheus queries and merge them manually.

Thanos Query is a stateless service that discovers Prometheus instances and other Thanos sidecars, queries them in parallel, and deduplicates results. It maintains awareness of which replica scraped which metrics, so even if you run three identical Prometheus instances for high availability, Thanos knows not to triple-count your metrics.

Let's say you deploy Thanos Query with a service discovery config:

--query.replica-label=replica
--query.partial-response
--store.sd-dnssrv-interval=30s
--store.sd-dns-resolver=127.0.0.11:53

The replica-label flag tells Query to treat different scrape replicas as equivalent sources and intelligently choose one, avoiding duplication. The partial-response flag means if one cluster is temporarily unreachable, Query still returns data from the others—critical for reliability.

You'd point your Grafana datasource at Thanos Query, and suddenly all your dashboards work across clusters. A query like:

max(rate(container_cpu_usage_seconds_total[5m])) by (pod_name)

now executes against every Prometheus globally, and you see results from all of them. Mimir provides the same capability built-in; you configure your Grafana to point to the Mimir query frontend instead of individual Prometheus instances.

This shifts your operational mental model. Instead of "I have three Prometheus instances," you now think "I have one logical monitoring system with multiple ingestion points." Alerting, dashboarding, and troubleshooting all work against this unified view.

Downsampling for long ranges

Storing full-resolution metrics forever is wasteful. A production metric point every 15 seconds adds up quickly across millions of series. Thanos and Mimir both support downsampling: automatically creating coarser-grained versions of metrics as they age.

After, say, 48 hours, Thanos can downsample your 15-second resolution data to 1-minute resolution, then after 30 days to 5-minute resolution. When you query a 90-day range, the system automatically serves the 5-minute version instead of trying to stream 360,000 points per series.

This happens transparently. You configure it like so:

compact:
  enable-vertical-compaction: true
  block-ranges:
    - 1h
    - 24h
  downsample:
    disable: false

Thanos Compact (a background service) periodically processes blocks in object storage, merging smaller blocks together and creating downsampled versions. A block that originally contains 15-second data points gets rewritten to also contain 1-minute and 5-minute aggregates. When Query needs to answer a six-month query, it reaches for the 5-minute version automatically.

The beauty is that you don't write different queries for different ranges. Your PromQL stays the same; the system is intelligent about which resolution serves the query efficiently. Some teams keep full resolution for 7 days, 1-minute resolution for 30 days, and 5-minute for a year, achieving massive storage savings while still being able to investigate recent incidents with full detail.

Mimir includes similar downsampling, configured in the ruler or via the admin API. The key principle is the same: coarser resolution for older data, full resolution for recent data, all automated.

High availability

Traditional Prometheus is a single point of failure. Mimir and Thanos both solve this by distributing data across multiple replicas and using quorum-based reads and writes.

With Thanos, you run multiple Prometheus instances scraping the same targets (that's why we set replica: 0 and replica: 1 in the external labels earlier). Each Prometheus sends its blocks to the same object store. Thanos Query, when asked a question, queries both replicas and deduplicates based on the replica label. If one Prometheus dies, Query still answers from the other. If both die, you lose current data but your historical blocks remain in object storage untouched.

Mimir goes further. It uses a distributed hash ring to automatically shard incoming metrics across multiple ingesters. When you send a metric to Mimir's distributor, it routes the write to RF (replication factor) number of ingesters. If one dies mid-write, your metric still gets persisted to the others. This is closer to Cassandra or Bigtable's model: not just redundancy, but genuine distributed data.

For both systems, object storage itself becomes your source of truth. Local disks are ephemeral. If a Prometheus or Mimir replica dies, its local disk is discarded, and the process comes back up and downloads blocks from object storage. This makes scaling trivial: spin up new instances, they hydrate from S3, and join the cluster.

A production Mimir cluster looks like:

distributor:
  rate_limit_enabled: true
  rate_limit_strategies: "global"
  rate_limit: 100000

ingester:
  lifecycler:
    ring:
      kvstore:
        store: consul
      replication_factor: 3

storage:
  engine: blocks
  backend: s3
  s3:
    bucket_name: metrics-prod

The distributor receives writes, the ingester stores them with RF=3, and object storage is the durable layer. A node failure is noise; the system keeps running.

Common mistakes

Forgetting object storage replication. Object storage like S3 is already replicated across zones, but some teams use MinIO on a single node. A single-node MinIO failure is a total outage. Always run MinIO with at least three nodes and proper erasure coding.

Configuring replica labels incorrectly. If you run multiple Prometheus replicas but don't set --query.replica-label=replica or forget to add the external label, Thanos Query will double-count or miss metrics. Verify your setup returns the expected cardinality.

Underestimating object storage latency. Queries spanning long ranges or many blocks now depend on S3 performance, not local disk. A slow object store makes your entire system feel sluggish. Monitor object storage latency as a critical metric.

Not tuning downsampling windows. If you downsample too aggressively (e.g., 5 minutes after 24 hours), you lose granularity when troubleshooting recent incidents. Start conservative: keep full resolution for at least 7 days.

Key takeaways

• Object storage decouples Prometheus disk capacity from your retention window, allowing unlimited history at predictable cost.
• Thanos is an add-on to existing Prometheus; Mimir replaces it entirely. Both solve the same problem with different trade-offs.
• A unified query layer across multiple clusters eliminates manual metric merging and makes global alerting possible.
• Downsampling automatically tunes storage efficiency based on metric age, keeping queries fast without sacrificing historical detail.
• Distributed replication and object storage as the source of truth make these systems resilient to node failures and trivial to scale.