Histograms and percentiles: a practical Monitoring guide

Most teams measure API latency with a single average, then wonder why their dashboards show "200ms average" while customers complain about 5-second request hangs. The problem isn't your application—it's your metrics. Histograms and percentiles: a practical Monitoring guide explains why you need to measure latency correctly, and how Prometheus histograms let you capture the distribution of response times rather than collapsing them into a meaningless mean.

Why averages lie and distributions tell the truth

Before we talk about histograms, let's establish why we need them. Consider a payment processing API where 99% of requests complete in 100ms, but 1% timeout at 30 seconds. The average latency might be 400ms—technically accurate, but completely useless for operations. Your p50 (median) is 100ms. Your p99 is 30 seconds. These are wildly different stories, and your SLO probably cares about p99, not the mean.

Prometheus counters and gauges can't capture this distribution. You need histograms. A histogram buckets observations into ranges and counts how many fall into each bucket. This lets you compute percentiles—rough estimates that show you where the bulk of your traffic sits and where your tail latencies lurk.

Buckets — Pick buckets around your SLO thresholds

The first decision when instrumenting a histogram is bucket configuration. Prometheus histograms use cumulative buckets: each bucket represents "requests faster than or equal to X milliseconds," and the counts are cumulative. If your buckets are 10ms, 50ms, 100ms, 500ms, 1s, and you have 5 requests totaling 200ms, 150ms, 80ms, 600ms, and 1200ms, the bucket counts would be:

• 10ms bucket: 0 requests (nothing ≤ 10ms)
• 50ms bucket: 1 request (one request ≤ 50ms)
• 100ms bucket: 2 requests (two requests ≤ 100ms)
• 500ms bucket: 3 requests (three requests ≤ 500ms)
• 1s bucket: 3 requests (still three)
• +Inf bucket: 5 requests (all requests included)

Here's the critical principle: pick buckets around your SLO thresholds. If your SLO states "95% of requests complete within 200ms," you want buckets at or just above 200ms. If you have tiers—gold customers get 50ms SLA, silver get 200ms—bucket at both 50ms and 200ms.

In your application instrumentation, it might look like this:

http_request_duration_seconds = Histogram(
  'http_request_duration_seconds',
  'HTTP request duration',
  buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0),
  labelnames=['method', 'endpoint', 'status']
)

These buckets (5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s) are reasonable defaults for web services. The key is that 250ms and 500ms exist—if your SLO is 400ms, you now have good resolution around that threshold. Some teams use exponential bucket schemas; newer Prometheus versions (2.40+) support EXP_BUCKETS to auto-generate buckets with exponential spacing.

histogram_quantile — Estimates percentiles from buckets

Once you have histogram data flowing into Prometheus, you compute percentiles with the histogram_quantile function. It takes a quantile (0.5 for p50, 0.99 for p99) and a histogram metric, then estimates the percentile based on linear interpolation within buckets.

In Grafana, a query to fetch p99 latency for your API might look like:

histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

This reads as: "From the rate of observations in each bucket over the last 5 minutes, estimate the value below which 99% of requests fall." The result is a time series you can graph, alert on, or use in SLO dashboards.

Let's say you want to track p50, p95, and p99 for a specific endpoint:

histogram_quantile(0.50, rate(http_request_duration_seconds_bucket{endpoint="/api/checkout"}[5m]))
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{endpoint="/api/checkout"}[5m]))
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{endpoint="/api/checkout"}[5m]))

Each query gives you a separate line on your dashboard. The limitation: histogram_quantile is an estimate. If most of your requests fall in the 100–500ms bucket and none in the 50ms bucket, the p99 calculation assumes uniform distribution within that 100–500ms range. If your real p99 is 250ms, histogram_quantile might report 280ms. For most SLO monitoring, this accuracy is sufficient. For extreme precision (e.g., financial microsecond trading), you'd use native histograms.

Aggregate first — Sum buckets across instances before computing quantiles

Here's a trap many teams fall into: they compute histogram_quantile on individual instances, then average those percentiles. This is wrong.

Wrong approach:

avg(histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) by (instance))

This computes p99 for each instance separately, then averages the results. If instance A has p99 of 150ms and instance B has p99 of 500ms (due to load imbalance), the average is 325ms—but that doesn't reflect the true p99 across your fleet.

Correct approach: aggregate buckets first, then compute quantiles.

histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Here, sum(...) by (le) combines the bucket counts across all instances. The le label represents the bucket upper bound (e.g., "100ms", "500ms", "+Inf"). Once buckets are summed, histogram_quantile computes the p99 across your entire fleet. This is the correct way to think about fleet-wide latencies.

In your dashboard provisioning or alert rules, always follow this pattern:

- alert: HighLatencyP99
  expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
  for: 5m
  annotations:
    summary: "P99 latency exceeds 500ms"

This fires when the fleet-wide p99 latency exceeds 500ms for 5 minutes—a real signal that users are experiencing slowdowns, not a false alarm from a single overloaded instance.

Native histograms — Newer Prometheus versions support high-resolution histograms

Prometheus 2.40 introduced native histograms, a new format that solves the bucketing problem. Instead of pre-defining buckets, native histograms store observations at high resolution and bucket them dynamically at query time. They use exponential bucket schemas, similar to OpenTelemetry's approach.

The advantage: you no longer need to guess bucket sizes. If you instrument a histogram with native histogram support, Prometheus ingests every observation and can answer arbitrary quantile queries without information loss from bucketing.

From the instrumentation side (if your client library supports it), you'd enable native histograms:

http_request_duration_seconds = Histogram(
  'http_request_duration_seconds',
  'HTTP request latency',
  native_histogram=True
)

On the Prometheus config side, you enable support:

global:
  feature_flags:
    - native-histograms

The trade-off: native histograms consume more storage and ingestion resources. For most teams, classic buckets with thoughtful SLO-aligned thresholds are sufficient. Native histograms shine when you have hundreds of metrics, changing SLOs, or when you need extremely precise tail latency measurements. If you're running Prometheus 2.40+, it's worth experimenting with a subset of critical histograms to see if the accuracy gain justifies the cost.

Common mistakes

• Computing quantiles per-instance then averaging: You lose the global picture. Always aggregate buckets first with sum(...) by (le), then compute quantiles.
• Buckets too coarse or too fine: Buckets at 1ms intervals for a service with 200ms SLO are useless noise. Buckets at 1-second intervals when your SLO is 100ms miss the detail. Align buckets to your SLOs and expected latency ranges.
• Forgetting the +Inf bucket: Prometheus adds this automatically, but if you're importing histograms or debugging, remember it's required. It represents the total request count.
• Using histogram_quantile on raw histogram metrics without rate(): Always use rate() or increase() to convert cumulative bucket counts into rates over a time window. Raw buckets are monotonic and will give nonsensical results.
• Not labeling by endpoint or operation: A single "http_request_duration_seconds" histogram for all traffic hides slow endpoints behind fast ones. Use labels like endpoint, operation, or method to break down latency by service tier.

Key takeaways

• Histograms capture request latency distributions, not just averages. Percentiles (p50, p95, p99) tell you where real customers experience slowdowns.
• Choose bucket boundaries around your SLO thresholds—50ms, 100ms, 200ms, 500ms, 1s—so you have good resolution where it matters.
• Use histogram_quantile(quantile, rate(metric_bucket[window])) to estimate percentiles from buckets; it's an approximation, but sufficient for SLO monitoring.
• Always aggregate buckets across instances with sum(...) by (le) before computing quantiles. Computing per-instance quantiles then averaging is incorrect.
• Native histograms (Prometheus 2.40+) remove the bucketing guesswork, but use more storage. For most teams, thoughtful classic buckets are the pragmatic choice.