Prometheus interview questions for Monitoring developers

Introduction

Whether you're preparing for an SRE or DevOps interview, or you're the one asking the questions, Prometheus interview questions for monitoring developers reveal how well someone understands metrics collection, time-series databases, and production observability. We've seen candidates with years of experience stumble on cardinality trade-offs, while newer engineers grasp the pull model instantly. This guide covers the four foundational concepts that separate those who configure Prometheus from those who truly operate it at scale.

Over our time building and scaling monitoring systems across e-commerce, SaaS, and trading platforms, we've learned that the right questions cut through resume noise. They expose whether someone has actually debugged a cardinality explosion at 3 am, or just read the documentation. Let's walk through the questions you should know cold.

Pull vs Push: Why Prometheus Pulls and What It Means

The first and most fundamental question in any Prometheus interview is simple: does Prometheus push or pull metrics? The answer—Prometheus pulls—unlocks understanding of its entire architecture.

Prometheus uses a pull model. Every scrape interval (default 15 seconds), Prometheus connects to targets and scrapes the /metrics endpoint. This is the opposite of systems like Graphite or InfluxDB, where agents push data to a central server. The distinction matters more than it sounds.

Why Pull Is Simpler

Pull-based collection simplifies service discovery and health detection. Consider a microservice that crashes: with a push model, you need the agent to notice and stop sending, or you get false signals. With pull, Prometheus tries to scrape, the connection fails, and you immediately know the target is down. No agent reliability problem. No heartbeat timeouts to tune.

Service discovery becomes stateless too. Add a new Kubernetes pod, and Prometheus finds it via service discovery rules—no agent configuration needed. Remove it, and Prometheus stops scraping. The target list is declarative and self-healing.

Real-World Example: Database Failover

Imagine your database primary fails and a replica promotes. With push-based metrics:

• The old primary's agent might keep pushing stale data for 30 seconds (timeout window).
• The new primary's agent might take time to warm up and start pushing.
• You see a gap or overlap in the time series.

With Prometheus's pull model:

• Scrape fails on the old primary—you see up=0 instantly.
• The new primary's port comes online, DNS updates, Prometheus scrapes it next cycle.
• Your alerting rules respond to up changes immediately.

This is why Kubernetes ecosystems standardized on Prometheus. The pull model aligns perfectly with dynamic infrastructure.

Cardinality: The Silent Killer at Scale

Now the harder question: what is cardinality, and why does it matter? Cardinality is the number of unique time series in your Prometheus instance. It's the single largest scaling constraint you'll hit.

A time series is uniquely identified by its metric name and label combination. If you have a metric http_requests_total with labels method, endpoint, and status, and those labels have 5, 50, and 3 unique values respectively, you have 5 × 50 × 3 = 750 series just for that one metric. Multiply that across hundreds of services and you can easily hit tens of millions of series.

Why It Kills You

Each time series consumes RAM, index memory, and disk I/O. At 10 million series, a single Prometheus instance needs 64+ GB of RAM just to keep the index hot. Query performance degrades. Cardinality is the reason we see monitoring stacks collapse—not because they lack clever alerting, but because they chose labels poorly.

Common cardinality traps include:

• High-cardinality identifiers as labels: Using request IDs, user IDs, or transaction IDs as label values. Each unique value creates a new series.
• Timestamps as labels: Putting a timestamp field in labels defeats Prometheus's time-series model entirely.
• Unbounded service lists: If you label every request with the backend service it hit, and you have 500 microservices, you've created a cardinality multiplier.

Detecting and Fixing Cardinality Problems

When cardinality spirals, use this query to find the worst offenders:

topk(10, count by (__name__, job) ({__name__=~".+"}))

This lists the 10 metrics × job combinations with the most series. From there, examine individual metrics:

topk(5, count by (instance, method, endpoint) (http_requests_total))

Often you'll find a label that should be dropped entirely, or one that should be moved to an exemplar or separate span.

The fix is relabeling. Use metric_relabel_configs to drop high-cardinality labels at scrape time:

scrape_configs:
  - job_name: 'myapp'
    metric_relabel_configs:
      - source_labels: [request_id]
        action: drop

Or use keep to be explicit about what you want to retain. Always default to dropping labels you don't alert or dashboard on.

Computing p95 Latency: histogram_quantile and Rate Semantics

A strong interview candidate should be able to write a query to compute p95 latency from a Prometheus histogram. This tests understanding of histograms, bucketing, and PromQL semantics.

Start with the metric definition. When your application records request latencies, it emits a histogram:

request_duration_seconds_bucket{le="0.001"}
request_duration_seconds_bucket{le="0.01"}
request_duration_seconds_bucket{le="0.1"}
request_duration_seconds_bucket{le="1"}
request_duration_seconds_bucket{le="+Inf"}
request_duration_seconds_sum
request_duration_seconds_count

Each bucket is cumulative. The le="0.01" bucket includes all requests with latency ≤ 10ms. To compute a percentile, Prometheus interpolates between buckets.

The Query and Why It Works

To get p95 latency over the last 5 minutes, averaged per endpoint:

histogram_quantile(0.95, 
  sum by (le, endpoint) (
    rate(request_duration_seconds_bucket[5m])
  )
)

Here's what each part does:

• rate(): Converts bucket counts to a rate (requests/sec). This makes the quantile calculation work across time ranges correctly. Without rate(), histograms from different scrape intervals give wrong results.
• sum by (le, endpoint): Aggregates buckets across all instances while keeping the bucket boundary (le) and endpoint label. This is critical—you must preserve le or histogram_quantile has no buckets to interpolate.
• histogram_quantile(): Interpolates to find the 0.95 threshold (95th percentile).

A common mistake is forgetting rate() or summing away the le label. Both break the query silently and give wrong numbers.

Multi-Percentile Dashboard Card

In practice, you'll want p50, p95, and p99 together:

histogram_quantile(0.50, sum by (le, endpoint) (rate(request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum by (le, endpoint) (rate(request_duration_seconds_bucket[5m])))
histogram_quantile(0.99, sum by (le, endpoint) (rate(request_duration_seconds_bucket[5m])))

Notice p50 and p99 are almost identical queries, just with different quantile values. Once you have the pattern right, it scales easily.

Alerting on SLOs: Multi-Window Burn Rate Rules

The final and most complex question: how do you alert on SLOs in Prometheus? A naive answer is "alert when availability drops below 99.9%." A strong answer involves burn rate, time windows, and alert latency trade-offs.

SLOs are typically defined as a target error rate over a period. If your SLO is 99.9% availability over 28 days, you can tolerate roughly 40 minutes of downtime per month. The burn rate is how fast you're consuming that budget. A 100% error rate burns at 1440× the allowed rate.

The Multi-Window Approach

Google's SRE book recommends multi-window burn rate alerts. Use two windows: one fast (e.g., 5 minutes) and one slow (e.g., 1 hour). Fire an alert only if both are above threshold, adjusted for the window size.

Example: SLO is 99.9% over 28 days. In 5 minutes, you can "spend" (1 - 0.999) × (5 / (28 × 24 × 60)) of your budget, about 0.025%. In 1 hour, about 0.3%.

If your 5-minute error rate is above 0.25% (10× the allowed burn rate), that's bad. If your 1-hour error rate is also above 3% (10× the allowed burn rate for an hour), you're in a real incident.

groups:
  - name: slo_burn_rate
    interval: 30s
    rules:
      - alert: SLOBurnRateHigh
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.0025
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > 0.003
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "SLO burn rate high"

This fires if both conditions are true for 2 minutes. The for clause prevents noise from brief blips. The dual-window approach catches real problems without false alarms.

Why This Matters in Production

We've seen teams with "availability below 99.9%" alerts fire constantly, then get ignored. The multi-window approach is better: it signals when you're actually at risk of breaching SLO. It gives you time to respond without crying wolf every time there's a 30-second blip.

Common Mistakes

Confusing scrape interval with resolution: Scrape interval (how often you collect) is not the same as resolution (how often you can query). A 15-second scrape interval doesn't mean you can query at 1-second granularity. The data doesn't exist.

Forgetting rate() in aggregations: When you sum or average histogram buckets across instances, use rate() first. Otherwise, your bucket counts are absolute values, not rates, and histogram_quantile produces nonsense.

Using presence alerts for reliability: Alerting only on up == 0 is reactive. A target can be up but returning errors. Pair it with error-rate alerting to catch real issues before your users do.

Setting cardinality-unaware label schemas: It's tempting to label everything—endpoint, method, status, instance. But 10 labels × 100 values each = 10 billion series (in principle). Start minimal, add labels only when you query them regularly.

Tuning alert thresholds without baseline: Alerting on "CPU above 70%" without knowing your normal is asking for noise. Use percentiles of historical data, or set the threshold based on SLO burn rate like above.

Key Takeaways

• Prometheus's pull model simplifies service discovery and health detection compared to push-based systems; targets fail cleanly and there's no agent state to manage.
• Cardinality—the number of unique time series—is the primary scaling limiter; design labels deliberately and drop high-cardinality fields like request IDs or timestamps.
• Computing percentiles requires rate() to normalize bucket counts across time ranges, and you must preserve the le label so histogram_quantile can interpolate.
• SLO alerting should use multi-window burn rate rules to catch real incidents without noise; a single-window threshold alert fires constantly and trains teams to ignore it.
• In interviews, expect questions to dig into why things work the way they do, not just how; be ready to explain trade-offs and production failure modes.