Monitoring multi-region systems: lessons from production

Monitoring multi-region systems: lessons from production

When your platform spans multiple geographic regions, a single Prometheus instance becomes a bottleneck. We've learned the hard way that visibility across regions requires deliberate architecture decisions—from how you label metrics to how you query them globally. This article covers the practical patterns we've used to monitor distributed systems serving traffic from North America, Europe, and Asia without losing observability.

Why region isolation creates blind spots

Early in our multi-region journey, we ran independent Prometheus instances per region. Each collected metrics from its local services, databases, and infrastructure. This worked until we needed to answer simple questions: "Which region has the highest error rate?" or "Are we experiencing a global database connection pool problem?"

The answer required logging into three different Grafana instances, navigating to the same dashboard three times, and manually comparing numbers. Within months, we had dashboards diverging, alerts configured differently per region, and blind spots where a subtle issue affected multiple regions but wasn't obvious from isolated views.

Region labels on every series

The foundation of multi-region monitoring is enforcing a consistent labeling strategy. Every metric must carry a region label that identifies where it originated. This isn't optional—it's the contract that makes global queries possible.

In practice, we add the region label at scrape time using relabel configs or at the application level using client libraries. Here's how we configure it in Prometheus:

global:
  external_labels:
    region: us-east-1
    cluster: prod-primary

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          region: us-east-1

The external_labels section broadcasts the region to every series Prometheus scrapes. We do this for each regional Prometheus instance, setting the appropriate region value. We also add it explicitly in per-target scrape configs as a safety net.

Application-level labels work when services emit metrics directly. Using Prometheus client libraries:

from prometheus_client import Counter

request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    labelnames=['region', 'method', 'status']
)

request_count.labels(
    region='eu-west-1',
    method='GET',
    status='200'
).inc()

Every metric now includes the originating region. This becomes your query dimension. When you write http_requests_total{region="us-east-1"}, Prometheus returns only US data. Write http_requests_total without filtering, and you get all regions combined—ready for aggregation.

Global views with Thanos or Mimir

Having region labels is useless if you can't query across regions easily. This is where you need a solution that aggregates metrics from multiple Prometheus instances into a queryable store. The two main options in production are Thanos and Grafana Mimir.

Thanos is a set of components that sit alongside your existing Prometheus instances. It reads data from Prometheus via the remote read API, deduplicates it, stores it in object storage (S3, GCS, Azure Blob), and exposes a query interface. You deploy a Thanos Querier that talks to all your regional Prometheus instances simultaneously.

Setup looks like this:

# Each regional Prometheus sends blocks to object storage
objstore_config:
  type: s3
  config:
    bucket: monitoring-blocks-prod
    endpoint: s3.amazonaws.com

# Thanos Querier knows about all regional Prometheus instances
querier:
  stores:
    - dnssrv+_grpc._tcp.prometheus-us-east-1.monitoring:10901
    - dnssrv+_grpc._tcp.prometheus-eu-west-1.monitoring:10901
    - dnssrv+_grpc._tcp.prometheus-ap-south-1.monitoring:10901

When you query Thanos, it fan-outs to all stores, collects results, deduplicates by timestamp and labels, and returns unified data. A single sum(rate(http_requests_total[5m])) by (region) query now shows you request rates per region in one result set.

Grafana Mimir is a managed-feeling solution. You push metrics from regional Prometheus instances to a central Mimir cluster using the remote write API. Mimir handles storage, deduplication, and querying. It's simpler operationally—no need to manage object storage separately or run Thanos sidecars.

# In each regional Prometheus config
remote_write:
  - url: https://mimir-prod.monitoring.internal/api/prom/push
    write_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_.*|http_.*|postgres_.*'
        action: keep

Metrics land in Mimir's distributed store. Query Mimir's Prometheus-compatible API from Grafana, and you get the same multi-region perspective. The trade-off: Mimir requires push-based architecture, which means higher network overhead for high-cardinality metrics, but gains easier horizontal scaling.

We use Thanos in systems with very high cardinality where push overhead becomes problematic, and Mimir where simplicity matters more than optimization.

Compare regions side by side

Now that you have unified metric storage, dashboard design becomes critical. A good multi-region dashboard shows regional comparisons clearly without requiring manual flipping between views.

We use variable-based dashboards with Grafana's multi-select region variable:

# Grafana variable definition
{
  "name": "region",
  "type": "query",
  "datasource": "Prometheus",
  "query": "label_values(http_requests_total, region)",
  "multi": true,
  "includeAll": true
}

A single panel then filters by the selected regions:

sum(rate(http_requests_total[5m])) by (region)
# With variable substitution:
sum(rate(http_requests_total{region=~"$region"}[5m])) by (region)

But the real power comes from side-by-side comparisons. Create a table that shows key metrics per region:

This table is built with Grafana's table panel and a query that groups metrics by region, making anomalies obvious. If one region suddenly shows 5x higher latency, you see it immediately.

Annotation-based comparisons help too. Add regional events—deployments, scaling events, database failovers—as annotations. A timeline showing when each region deployed a new version makes correlation obvious when issues appear.

Probes from each region

Synthetic monitoring from each region catches issues that regional metrics alone miss. A service might be healthy according to internal metrics but unreachable from certain geographies due to network routing, firewall rules, or CDN misconfiguration.

Prometheus Blackbox Exporter, deployed in each region, probes your critical endpoints:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      follow_redirects: true

  tcp_connect:
    prober: tcp
    timeout: 5s

scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://api.example.com/health
          - https://checkout.example.com/status
        labels:
          region: us-east-1
      - targets:
          - https://api.example.com/health
          - https://checkout.example.com/status
        labels:
          region: eu-west-1
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

Each regional Blackbox exporter probes the same endpoints. Differences in response times or success rates reveal geographic latency issues, regional outages, or routing problems. Query probe_duration_seconds{region="ap-south-1", job="api-health"} and you know if that region's users experience slow API responses.

Combine probes with alerts that fire only when multiple regions are affected. A single region's brief blip doesn't page you; a coordinated issue across two or more regions does.

Common mistakes

• Missing region labels on some metrics. If infrastructure metrics lack region labels but application metrics have them, you can't correlate. Audit your metric output regularly and add region labels everywhere, not just "most places."
• Inconsistent region naming. One service calls it "us-east-1", another calls it "useast1". This breaks queries and aggregations silently. Standardize on one format per organization and enforce it in code reviews.
• Global aggregations without regional context. Summing http_requests_total across all regions hides problems. A region handling 10% of traffic but generating 50% of errors is invisible in global sums. Use by (region) to preserve regional visibility.
• Alerting only on global metrics. An alert firing when global error rate exceeds 1% doesn't tell you which region to investigate. Alert on regional metrics: error_rate{region=~"us-east-1|eu-west-1"} > 0.02.
• Forgetting to test failover scenarios. Your multi-region setup looks good until a regional Prometheus dies and suddenly queries break or return partial data. Test what happens when one region goes silent.

Key takeaways

• Add region labels to every metric at scrape time or application level—it's the foundation for multi-region queries.
• Use Thanos for pull-based aggregation of independent Prometheus instances or Mimir for push-based centralization, depending on your cardinality and operational complexity.
• Build Grafana dashboards that show regional comparisons in tables and side-by-side panels; avoid forcing operators to query three separate instances manually.
• Deploy synthetic probes (Blackbox Exporter) from each region to catch geographic reachability and latency issues that internal metrics miss.
• Alert on regional metrics, not just global aggregates, so incidents are actionable and traceable to a specific geography.