PromQL examples you will use daily: a step-by-step guide

Getting started with PromQL in production

You've set up Prometheus, scraped metrics from your services, and now you're staring at the query interface wondering what to actually type. PromQL examples you will use daily: a step-by-step guide will take you through the queries that appear in nearly every monitoring setup—the ones you'll find yourself writing repeatedly across different services and dashboards. These aren't theoretical exercises; they're the building blocks of real alerting rules and dashboards that keep teams aware of what's happening in production.

Let's walk through the four categories of queries you'll reach for constantly: request rates, error ratios, latency percentiles, and target health. By the end, you'll have a mental model for why each query works and when to use it.

Request rate: measuring traffic per second

Understanding how much traffic your services handle is foundational. The request rate tells you whether load is stable, spiking, or dropping—critical information for diagnosing performance problems or confirming that a deployment had the intended effect.

The most common pattern is measuring requests per second over a rolling window:

sum(rate(http_requests_total[5m])) by (route)

Here's what each piece does:

• http_requests_total is a counter metric (always increasing) that tracks total requests since the service started.
• rate(...[5m]) calculates the per-second rate over the last five minutes, handling counter resets gracefully.
• sum(...) by (route) groups the results by the route label, so you see each endpoint's traffic separately.

In a real dashboard, you'd adjust the window based on your needs. A five-minute window smooths out second-to-second noise; a one-minute window is more responsive to sudden changes. If you're querying from a dashboard with a time range picker, you might use a variable: sum(rate(http_requests_total[$__interval])) by (route).

A practical scenario: you deploy a new service endpoint and want to confirm traffic is routing to it. You'd create a panel with this query, grouped by route, and watch the new endpoint's line appear as traffic begins flowing. If it stays at zero, you know something's wrong with routing or the service isn't healthy.

Another common variant includes filtering by method or status:

sum(rate(http_requests_total{method="POST"}[5m])) by (route)

This narrows the view to POST requests only, useful when you're investigating a specific type of operation.

Error ratio: identifying failing services

Raw error counts are misleading—a service handling 100 requests per second with 5 errors looks very different from one handling 1 request per second with 1 error. The error ratio normalizes this by expressing errors as a percentage of total requests.

Here's the standard pattern:

sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

Breaking this down:

• {status=~"5.."} uses a regex to match all 5xx status codes (500, 501, 502, etc.).
• The numerator calculates the rate of errors.
• The denominator calculates the rate of all requests.
• Dividing gives you the error ratio as a decimal (e.g., 0.02 means 2%).

In a Grafana dashboard, you'd often multiply by 100 and format as a percentage: (sum(rate(...{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))) * 100.

You might also segment by service or route:

sum(rate(http_requests_total{status=~"5.."}[5m])) by (service) / on(service) sum(rate(http_requests_total[5m])) by (service)

The on(service) modifier tells Prometheus to join the two time series on the service label, so each service's error count is divided by that same service's total count.

A real-world use case: after a deploy, error ratio spikes from 0.1% to 3%. This query immediately tells you the deploy is problematic. Compare it against the baseline from yesterday, and you have clear evidence to roll back.

You might also want to track 4xx errors separately (client errors) since they're typically less urgent:

sum(rate(http_requests_total{status=~"4.."}[5m])) / sum(rate(http_requests_total[5m]))

Latency p95: histogram quantiles for performance

Request rate and error ratio tell you about volume and reliability; latency tells you about user experience. The 95th percentile (p95) is the standard threshold—it means 95% of requests finished faster than this value, and only 5% were slower. This filters out occasional spikes while catching genuine performance problems.

If your service exports a histogram metric, the query looks like:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Histograms work differently than counters. Instead of a single value, they have multiple buckets representing ranges (e.g., 0.001s, 0.01s, 0.1s, 1s). Prometheus uses these buckets to estimate percentiles.

• http_request_duration_seconds_bucket is the histogram metric itself. The _bucket suffix is automatic.
• rate(...[5m]) converts the raw bucket counts into per-second rates.
• by (le) groups by the bucket boundary. le stands for "less than or equal to."
• histogram_quantile(0.95, ...) interpolates the p95 value from the buckets.

Typically, you'll want this broken down by route or service:

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))

A practical scenario: your payment endpoint's p95 latency has climbed from 200ms to 800ms. This query pinpoints which endpoint is slow. You can then investigate whether it's a database query, external API call, or compute-heavy logic.

You can also check multiple percentiles in a single dashboard panel to see the full distribution:

histogram_quantile(0.50, ...) # p50 (median)
histogram_quantile(0.95, ...) # p95
histogram_quantile(0.99, ...) # p99

Note: the accuracy of histogram quantiles depends on bucket sizing. If your buckets are too wide, you lose precision. If they're too narrow, you waste storage. A good practice is setting buckets that align with your SLA targets.

Target health: the "up" metric

Prometheus scrapes targets at regular intervals. If a target becomes unreachable, it's crucial to know immediately. The up metric tells you the health status of each scrape target.

A simple query:

up == 0

This returns only targets where up equals 0, meaning they're down or unreachable. In a dashboard alert, this might trigger if any target is down for more than a minute.

You can filter by job:

up{job="api-server"} == 0

Or use a logical NOT to see all healthy targets:

up == 1

A production scenario: one of your database monitoring endpoints goes down, but your database itself is still running fine. Without checking the up metric, you'd miss that your observability is blind to that instance. With it, you immediately know to investigate why the exporter isn't responding.

You can also combine it with other metrics to catch cascading failures. For example, alert if any service instance is down:

count(up{job="api-server"} == 0) > 0

Common mistakes

• Forgetting the range vector. rate(http_requests_total) without [5m] won't work. Always specify a time window.
• Mixing up rate and increase. rate() gives per-second values; increase() gives raw growth. For most dashboards, rate() is what you want.
• Division by zero. If a denominator can be zero (e.g., a brand-new service with no traffic), your error ratio query will break. Guard against it with a conditional: (errors / total) > 0 or vector(0).
• Ignoring bucket boundaries in histograms. If your histogram doesn't have a bucket that matches your SLA target, your percentile estimates will be inaccurate. Plan bucket sizes upfront.
• Using fixed windows instead of variables. Hardcoding [5m] works, but Grafana's $__interval variable adapts to your dashboard's time range, giving better results across different views.

Key takeaways

• Request rate with rate()` and rate with `[5m]` window gives you per-second traffic, grouped by relevant labels like route or service.
• Error ratio divides error rates by total rates to normalize across services of different scales; regex matching like status=~"5.." filters specific codes.
• Latency p95 using histogram_quantile() requires properly sized buckets and always needs the by (le) grouping to work correctly.
• The up metric reveals gaps in your observability itself; always monitor whether your monitoring targets are reachable.
• Start with these four query patterns and adapt them by adding or removing labels; most production dashboards are variations of these fundamentals.