Managed Prometheus on AWS: practical tips and trade-offs

Introduction

By day 52 of running Prometheus at scale, most teams have wrestled with the operational burden of keeping it alive—disk management, retention policies, high availability setup, and the constant worry about data loss. AWS offers a managed alternative that removes much of this toil, but it comes with its own set of trade-offs worth understanding.

In this article, we'll explore Managed Prometheus on AWS: practical tips and trade-offs, examining how to integrate remote write from your agents, what you gain by eliminating server management, how to query data via Amazon Managed Grafana, and what the per-sample and per-query pricing model actually means for your budget and architecture.

Remote Write from Your Agents

The foundation of Managed Prometheus is the remote write protocol, a long-established Prometheus feature that ships metrics to a remote backend rather than storing them locally. Instead of maintaining a Prometheus server with local storage, your agents—Prometheus instances running in Kubernetes clusters, on EC2, or even on-premises—push samples to AWS.

Setting up remote write is straightforward. You configure your Prometheus scrape jobs as usual, then add a remote_write section to prometheus.yml:

global:
  scrape_interval: 15s
  external_labels:
    cluster: production
    env: prod

remote_write:
  - url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123/api/v1/write
    queue_config:
      capacity: 10000
      max_shards: 200
      min_shards: 1
      max_samples_per_send: 500
      batch_send_wait_ms: 200
      min_backoff: 30ms
      max_backoff: 100ms
    write_relabel_configs:
      - source_labels: [__name__]
        regex: "temp_.*"
        action: drop

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: "node"
    static_configs:
      - targets: ["10.0.1.50:9100"]

A few details worth noting: the queue_config parameters control how Prometheus batches and retries writes. If your connection to AWS drops, the local queue buffers samples (up to capacity) until the endpoint recovers. The write_relabel_configs let you drop noisy or unwanted metrics before they consume your quota, which is critical for cost control.

In production, we typically run two or three Prometheus instances per cluster, each with remote write enabled. This approach gives you redundancy without needing a centralized Prometheus server. Each agent independently ships data to Managed Prometheus, and AWS handles deduplication and consolidation on the backend.

One thing to verify: ensure your Prometheus version supports the remote write protocol version that AWS expects. Most modern versions (2.30+) work fine, but always check the AWS documentation for your specific workspace region.

No Servers to Manage

The headline benefit of Managed Prometheus is operational simplicity. You don't deploy, patch, or scale Prometheus servers. AWS handles storage, replication, backups, and upgrades transparently. For teams that have spent weeks debugging disk space issues or tuning retention policies, this is genuinely liberating.

In a self-managed setup, a typical problem sequence looks like this: your Prometheus disk fills up, alerting stops firing, you scramble to reduce retention, and in the worst case, you lose historical data. With Managed Prometheus, AWS enforces retention limits per workspace, and you only pay for what you store. There's no surprise disk exhaustion at 2 a.m.

High availability is another area where managed services shine. If you've built HA Prometheus with multiple replicas, remote storage backends, and Thanos coordination, you know the complexity. Managed Prometheus gives you that reliability out of the box. AWS replicates your data across availability zones, so a single AZ failure doesn't lose metrics.

That said, managed services introduce their own operational modes. You trade server management for API dependency management. If the AWS API throttles your workspace or if there's a regional outage, you can't quickly spin up an alternative. Most teams mitigate this by keeping a small local Prometheus instance running in parallel during the transition period, so you have a fallback if the managed service has issues.

Another consideration: cost optimization becomes a different problem. Instead of managing storage size and retention directly, you're managing metric cardinality and write frequency. If you accidentally start emitting a thousand new time series from a new service, your bill can spike. We recommend implementing cardinality checks and write rate alerts from day one.

Query with Amazon Managed Grafana

Amazon Managed Grafana is the natural companion to Managed Prometheus. Rather than running your own Grafana instance, you provision a managed workspace and connect it to your Prometheus workspace via a built-in data source.

The setup is minimal: create a Grafana workspace, add Managed Prometheus as a data source, and start building dashboards. The dashboard definition language remains the same—JSON or the visual editor—so your existing Grafana knowledge transfers directly.

Here's a typical Grafana data source configuration that you'd paste into a provisioning YAML:

datasources:
  - name: AMP
    type: prometheus
    url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
    access: proxy
    isDefault: true
    jsonData:
      httpMethod: GET
      sigv4Auth: true
      sigv4AuthType: default

The sigv4Auth parameter is key—it uses IAM roles to authenticate, so you don't store credentials in Grafana. The Grafana workspace's execution role must have permission to call the Managed Prometheus API.

In practice, Managed Grafana dashboards perform identically to self-managed Grafana with a Prometheus backend. Query performance depends on your workspace's resource allocation (AWS calls this "workspace capacity"), not the Grafana instance itself. Most teams find that a mid-tier workspace handles thousands of dashboard queries per minute without issue.

One workflow caveat: if you're migrating from self-managed Grafana, re-test your alert rules and recording rules. Some PromQL edge cases behave slightly differently depending on how the Prometheus API is implemented. We recommend exporting all dashboards from your old Grafana, importing them into Managed Grafana, and spot-checking a few critical queries side-by-side before decommissioning the old setup.

You can also continue using self-managed Grafana instances and point them at Managed Prometheus via its HTTP API. This hybrid approach is common during migrations or if you already have Grafana running and don't want to refactor it.

Pay Per Sample and Query

Pricing is where managed Prometheus requires careful attention. AWS charges for two dimensions: ingested samples and API queries.

Sample pricing: each data point you write to Managed Prometheus costs money. If you emit http_requests_total{job="api", instance="i-123", endpoint="/users"} every 15 seconds, that's 4 samples per minute per time series. With 5,000 active time series, you're pushing 20,000 samples per minute, or roughly 29 million samples per day. At current AWS pricing (around $0.30 per million samples in most regions), that's about $9 per day or $270 per month, just for ingestion.

Query pricing: every time Grafana (or any client) executes a PromQL query, AWS charges per query. A dashboard with 20 panels running on a 5-minute refresh rate incurs 240 queries per hour. With hundreds of users viewing dashboards, query costs can exceed ingestion costs if you're not careful.

To control costs, implement these strategies:

• Aggressive relabeling: drop unnecessary labels and metrics before remote write. Every label you drop reduces cardinality and sample count. Use write_relabel_configs to strip debug tags, request IDs, and other high-cardinality dimensions.
• Recording rules in your agents: pre-aggregate expensive queries. Instead of storing raw http_request_duration_seconds buckets, compute http_request_duration_seconds:p95:5m as a recording rule and ship that instead. This reduces the raw time series you ship to AWS.
• Caching in Grafana: use Grafana's query caching features to avoid redundant API calls. Set reasonable cache TTLs on dashboards that don't need real-time data.
• Cardinality monitoring: set up alerts on your Prometheus agents to warn when the number of active time series exceeds a threshold. A single misconfigured exporter can generate thousands of unexpected series.

A useful formula: estimate your monthly cost as (daily_samples_in_millions * 30 * 0.30) + (monthly_queries_in_millions * 0.30). For most teams, samples dominate the bill. Every 10% reduction in cardinality saves meaningful money.

Common Mistakes

• Shipping raw cardinality to AWS: many teams set up remote write and immediately regret it when their bill arrives. Always test relabeling rules locally before enabling remote write. Drop test metrics, debug metrics, and high-cardinality dimensions.
• Neglecting the retry queue: if your remote write endpoint is unstable, Prometheus will queue samples locally. But the queue has finite capacity (default 10,000 samples). If remote write stays down for hours, you'll lose recent data. Monitor remote write success rate and availability.
• Mixing local and remote storage accidentally: if you enable remote write but also keep local Prometheus storage, you might think you have redundancy when you actually have two independent copies with no coordination. Decide early: is your agent a stateless forwarder, or a local backup?
• Assuming managed Grafana is the only query client: if you run scripts, CI/CD pipelines, or third-party tools that query Prometheus directly, they'll incur query charges on Managed Prometheus. Budget for query volume from all sources, not just Grafana dashboards.
• Ignoring regional latency: remote write introduces a round-trip to AWS. In poorly connected networks or high-latency regions, this can cause write timeouts. Monitor your remote write latency and adjust timeout and backoff parameters as needed.

Key Takeaways

• Remote write is the mechanism that decouples your agents from a central Prometheus server; configure it carefully with relabeling rules to control costs and quality.
• Eliminating server management is the primary benefit, but it shifts operational focus to cardinality, write rates, and query patterns instead.
• Amazon Managed Grafana integrates seamlessly with Managed Prometheus via IAM authentication and provisioning, making dashboard migration straightforward.
• Per-sample and per-query pricing rewards efficiency; aggressive relabeling, recording rules, and query caching are essential to keep bills reasonable.
• Start with a parallel or shadow deployment of Managed Prometheus while keeping self-managed infrastructure; migrate dashboards and queries incrementally to catch surprises.
• Monitor your remote write queue depth, cardinality, and query volumes from day one; these metrics predict cost and reliability issues weeks before they become critical.