Prometheus metric types: real-world lessons and examples

By day five of your monitoring journey, you've set up Prometheus and learned the basics of scraping. But collecting metrics is only half the battle—choosing the right metric type for the job makes the difference between useful alerts and noisy dashboards. In this piece, we'll walk through Prometheus metric types: real-world lessons and examples, so you can instrument your services with confidence.

Prometheus gives us four metric types, each designed for a specific pattern. Pick the wrong one, and you'll spend hours trying to make sense of your data. Pick the right one, and your graphs tell the story of your system instantly.

Counter: only goes up — requests, errors

A counter is the simplest metric type: it only increases or stays the same. It never decreases (except during a restart). Think of it as an odometer in a car—it only goes forward.

Counters are perfect for tracking events: HTTP requests handled, errors logged, messages processed from a queue, database transactions committed. In production, we use counters obsessively because they're monotonic and thus extremely reliable for rate calculations.

Here's a real example from a payment processing service:

from prometheus_client import Counter

payment_processed = Counter(
    'payments_processed_total',
    'Total number of payments processed',
    ['currency', 'status']
)

# In your request handler:
payment_processed.labels(currency='USD', status='success').inc()
payment_processed.labels(currency='EUR', status='failed').inc()

Notice the _total suffix—this is a convention. Prometheus uses it to signal "this is a counter." When you query this metric, you'll typically apply the rate() function to see how many payments per second we're processing:

rate(payments_processed_total{status="success"}[5m])

This tells you the average success rate over the last five minutes. Because counters only go up, rate calculations are mathematically sound.

One gotcha: if you restart the application, the counter resets to zero. Prometheus handles this gracefully by detecting the reset and still computing correct rates, but be aware that looking at raw counter values is useless—always use rate(), increase(), or similar functions.

Gauge: goes up and down — memory, queue size

A gauge is the opposite of a counter. It can increase, decrease, or jump to any value. Use gauges for measurements that fluctuate: current memory usage, active database connections, items in a queue, temperature sensors, number of goroutines.

In a real ride-hailing system, we track active driver locations and passenger requests:

from prometheus_client import Gauge

active_drivers = Gauge(
    'drivers_active',
    'Number of drivers currently online',
    ['region']
)

passenger_queue = Gauge(
    'passenger_requests_waiting',
    'Passengers waiting for a match',
    ['region']
)

# Update every few seconds from your state store:
active_drivers.labels(region='us-west').set(1240)
passenger_queue.labels(region='us-west').set(87)

Unlike counters, you can query gauge values directly—they represent a snapshot. When you graph a gauge over time, you see the actual fluctuations:

drivers_active{region="us-west"}

You can also apply functions like avg_over_time() or max_over_time() to understand patterns:

avg_over_time(drivers_active{region="us-west"}[1h])

A common mistake is using gauges for things that should be counters. For instance, if you measure "total bytes sent" as a gauge, you'll lose the ability to calculate throughput accurately because gauges can go down (e.g., after a restart or bug). Always reach for counter for cumulative totals.

Histogram: distributions like latency

Histograms let you track the distribution of observations. Instead of storing every latency value, a histogram buckets them and counts how many fall into each bucket. This is invaluable for understanding tail latencies without storing gigabytes of raw data.

A histogram creates multiple time series automatically: cumulative bucket counts and a sum and count of all observations. For example, when you define a histogram for request latency, Prometheus automatically generates _bucket, _sum, and _count` metrics.

Here's how we instrument latency in a typical API:

from prometheus_client import Histogram

request_latency = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency in seconds',
    ['method', 'endpoint'],
    buckets=(0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0)
)

# In your middleware:
with request_latency.labels(method='GET', endpoint='/api/users').time():
    # Your handler logic
    pass

The buckets define the boundaries. Choose them based on your SLO. If your SLO says 99% of requests should complete in 100ms, include buckets like 0.05, 0.1, and 0.25 seconds so you can calculate the percentile accurately.

To get the 95th percentile latency, you query:

histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))

This is powerful. You get server-side quantile calculation, which means Prometheus does the math on your behalf. However, histograms are an approximation—the precision depends on your bucket boundaries. If you pick buckets too coarse, you lose detail. Too fine, and cardinality explodes.

In a CRM system handling millions of API calls daily, histograms let us track that 99th percentile latency per endpoint without storing individual timings. That's how we spot when search queries start taking 2 seconds instead of 200 milliseconds.

Summary: client-side quantiles, rarely needed

A summary also tracks distributions, but it computes quantiles on the client side (in your application) and ships the results to Prometheus. Unlike histograms, summaries don't use buckets—they're more precise but carry a higher computational cost and don't work well with aggregation.

Here's a summary example:

from prometheus_client import Summary

db_query_duration = Summary(
    'db_query_duration_seconds',
    'Database query duration in seconds',
    ['table']
)

# In your query handler:
with db_query_duration.labels(table='orders').time():
    result = db.query(sql)

Summaries automatically expose _sum, _count, and quantile metrics like db_query_duration_seconds{table="orders",quantile="0.95"}.

Why do we rarely use summaries in practice? Two reasons:

• No aggregation across instances. If you have ten application servers, each computing its own 95th percentile, you can't combine those percentiles meaningfully. With histograms, Prometheus aggregates buckets across all instances and computes the true 95th percentile.
• Higher cardinality. Every quantile becomes a separate time series label, doubling or tripling the cardinality.

Summaries have a place in very specialized scenarios—like tracking percentiles for something you absolutely cannot bucket. But in almost every case, histograms are the better choice. We recommend defaulting to histograms and only reaching for summaries if you have a specific reason.

Common mistakes

• Using a gauge for a total. "Total requests" should be a counter with _total suffix, not a gauge. Gauges reset when the app restarts, breaking rate calculations.
• Histogram buckets that are too coarse. If your SLO is 100ms but your smallest bucket is 1 second, you can't detect violations. Choose buckets that align with your SLOs.
• Forgetting that counters reset. Don't alert on raw counter values. Always use rate() or increase(). A reset looks like a drop to zero, which is fine—Prometheus handles it.
• Unbounded label cardinality. Never use user IDs, request IDs, or other high-cardinality values as metric labels. This will explode your Prometheus memory usage. Use low-cardinality labels: status codes, endpoints, regions, environments.
• Mixing histograms and summaries. Pick one for a given measurement. Mixing both creates confusion and doubles your cardinality.

Key takeaways

• Use counters for events and cumulative totals: requests, errors, messages processed. Always query with rate().
• Use gauges for measurements that go up and down: memory, queue size, active connections. You can query them directly.
• Use histograms for distributions like latency. They're server-side quantiles, work across multiple instances, and integrate cleanly with SLOs.
• Use summaries rarely. They're client-side, don't aggregate well, and add cardinality. Histograms are almost always the better choice.
• Pick low-cardinality labels and align histogram buckets with your SLOs. This keeps Prometheus healthy and your dashboards readable.