Common Prometheus mistakes: a practical Monitoring guide

Introduction

We've all been there: you wake up to a page because an alert fired, only to discover it was a false positive. Or worse, you miss a real incident because your monitoring was quietly collecting garbage data. After 76 days of building monitoring systems in production, we know that Prometheus is powerful—but it's also easy to misconfigure in ways that silently degrade the quality of your observability. Common Prometheus mistakes: a practical monitoring guide addresses the errors that hurt monitoring most, the ones that don't fail loudly but erode your confidence in alerting and make on-call exhausting.

In this article, we'll walk through four critical mistakes we've seen across e-commerce, CRM, SaaS, and trading platforms: high-cardinality labels that blow out storage, misusing rate() on gauges, alerts without for durations that create noise, and the statistical trap of averaging percentiles. Each mistake has a fix, and each fix is something your team can implement today.

High-cardinality labels: when flexibility becomes a liability

Labels in Prometheus are tempting. You can tag every metric with user_id, order_id, or request_id, and suddenly you have razor-sharp granularity. In theory, that sounds great. In practice, high-cardinality labels are one of the fastest ways to make Prometheus run out of memory and disk space.

Here's why: Prometheus stores every unique combination of metric name and label values as a separate time series. If you have a metric like http_requests_total with labels method, status, and user_id, and your application sees 10,000 unique users per day, you've just created 10,000 new time series per status code per HTTP method. On a busy system, this scales fast.

We once worked on a ride-hailing platform where engineers added driver_id as a label to a request latency histogram. Within a week, the Prometheus server ran out of disk space. The query performance tanked. The storage layer couldn't keep up. The fix took hours, and production observability was degraded the entire time.

The rule is simple: use labels for dimensions that are low-cardinality and stable. Good labels are things like method, status, region, or service. Bad labels are user IDs, request IDs, or transaction IDs—anything that grows unbounded.

If you need granular data for specific requests or users, use structured logging or a tracing system instead. Prometheus is for aggregated metrics, not per-entity tracking.

// DON'T DO THIS
http_requests_total{method="GET", status="200", user_id="12345"} 1
http_requests_total{method="GET", status="200", user_id="12346"} 1
http_requests_total{method="GET", status="200", user_id="12347"} 1
// ... millions of time series

// DO THIS INSTEAD
http_requests_total{method="GET", status="200"} 3
// Use logs or traces for per-user detail

Using rate() on gauges: a statistical error with real consequences

Gauges represent a value at a point in time—like memory usage, connections open, or queue depth. Counters only go up (or reset); they track cumulative totals like requests served or errors logged. This distinction matters because rate() only makes sense for counters.

When you call rate() on a gauge, you're asking Prometheus to calculate how fast the gauge is changing. Sometimes that works, but often it gives you nonsensical results. A gauge can go up and down randomly; calculating a rate of change on a gauge that bounces around tells you almost nothing useful.

We saw this in a CRM system where someone wrote an alert:

alert: MemoryLeak
expr: rate(memory_usage_bytes[5m]) > 1000000

This alert fires whenever memory changes by more than 1MB per 5 minutes. But memory is noisy. Garbage collection, cache fills, and temporary buffers cause legitimate spikes. The alert went off constantly. It was noise, not signal.

The correct approach depends on what you're measuring:

• For counters, use rate() to measure throughput or error rates.
• For gauges, look at absolute thresholds or use derivative functions sparingly and with clear intent.
• For histograms, use rate() on the _total suffix (which is a counter under the hood).

If you're trying to detect a memory leak, track the 95th percentile over a long window (hours) instead of the rate of change. That's stable, meaningful, and actionable.

// DON'T DO THIS
rate(memory_usage_bytes[5m]) > 1000000

// DO THIS INSTEAD
histogram_quantile(0.95, memory_usage_bytes) > 2000000000 // Alert if 95th percentile exceeds 2GB

Alerts without for durations: firing fast and breaking trust

By default, a Prometheus alert fires as soon as the condition is true. That sensitivity sounds good until your alerts are going off every time there's a blip in network latency or a temporary CPU spike during a deploy.

The for clause tells Prometheus to wait before firing. It acts as a debounce, ensuring the condition is consistently true for a period of time before waking up on-call. This single line prevents more noise than almost anything else we recommend.

In a SaaS platform we worked on, the team had an alert for request_latency_p99 > 500ms. No for clause. It fired dozens of times a day during peak traffic, even though the system was healthy and recovering on its own. On-call stopped trusting the alert. They stopped responding. Then a real incident happened and it took 20 minutes to notice.

The fix was a single addition:

alert: HighLatency
expr: histogram_quantile(0.99, request_duration_seconds_bucket) > 0.5
for: 5m  // Wait for 5 minutes before firing

Now the alert only fires if the p99 latency exceeds 500ms for at least 5 consecutive minutes. Temporary spikes are ignored. Real problems surface. On-call engineers trust the alert again.

Set for durations based on your system's recovery time. For a heavily cached system that recovers quickly, 2-3 minutes is reasonable. For a database alert or a deployment-related metric, 5-10 minutes makes sense. For critical availability, 1 minute is appropriate. The key is: pick a duration that matches your operational reality, not your ideal response time.

Averaging percentiles across instances: the statistical trap

Percentiles are powerful for understanding end-user experience. The p99 latency tells you how slow the slowest 1% of requests are. But percentiles don't average the way sums and rates do.

If you have 10 instances, each with a p99 latency of 100ms, the overall p99 latency is not 100ms. It's worse. The true p99 across all instances is higher because you're measuring the slowest 1% of all requests, not the average of the slowest 1% per instance.

In a trading platform, engineers were querying:

avg(histogram_quantile(0.99, rate(latency_bucket[5m])))

They were averaging the p99 from each instance. The result: a metric that was mathematically incoherent. It didn't represent the true p99 of the system. When they drilled into individual instances, latencies were much worse than the dashboard showed.

The correct approach is to let Prometheus aggregate the histogram buckets first, then calculate the percentile:

// DON'T DO THIS
avg(histogram_quantile(0.99, rate(latency_bucket[5m])))

// DO THIS INSTEAD
histogram_quantile(0.99, sum(rate(latency_bucket[5m])) by (le))

The second query aggregates the buckets across all instances, then calculates the percentile. That's the true p99 of your entire system.

If you need per-instance visibility, query each instance separately. If you need a system-wide view, aggregate the raw data, not the derived percentile.

Common mistakes: a quick reference

Key takeaways

• Cardinality is your enemy. Use labels for stable dimensions; use logs or traces for high-cardinality data.
• Match the operator to the metric type. rate() works for counters. Thresholds and careful derivatives work for gauges.
• Add for durations to every alert. A 2-10 minute buffer prevents noise and improves on-call reliability.
• Aggregate before you compute. Calculate percentiles from aggregated buckets, not from averaged percentiles.
• Test your alerts under load. Deploy your alerts to staging, generate realistic traffic, and watch them fire. Trust comes from confidence.

These mistakes are subtle because they don't fail loudly. Your Prometheus server keeps running. Dashboards keep rendering. But the data becomes unreliable, alerts become noise, and your team stops believing in monitoring. Fix these four things, and you'll have a monitoring system that actually earns the trust of your organization.