Business metrics in Prometheus for Monitoring developers

Introduction

By day 62 of your monitoring journey, you've likely mastered host metrics and application performance monitoring. But here's what separates teams that just react to outages from those that understand their business health: business metrics in Prometheus for monitoring developers bridge the gap between infrastructure dashboards and actual revenue impact. Tracking signups, orders, and revenue trends in the same system where you monitor CPU and latency gives your entire team a shared language for what matters.

When a spike in signup abandonment happens at 2 AM, wouldn't you rather wake up to an alert tied to business outcome rather than a vague "tail latency increased" notification? That's exactly what we'll explore today—how to instrument your applications to emit business events as Prometheus metrics, what pitfalls to avoid, and when to keep financial truth elsewhere.

Counters for Business Events

Prometheus counters are your primary tool for tracking business metrics. A counter only ever increases (or resets on service restart), making it ideal for monotonic events like signups, orders completed, or payments processed. Unlike gauges, which can fluctuate up and down, counters answer the question: "How many times has this thing happened since we started tracking?"

Let's say you're building an e-commerce platform. You'd instrument key business events across your checkout flow:

from prometheus_client import Counter

signup_total = Counter(
    'signups_total',
    'Total number of user signups',
    ['country', 'plan_type']
)

order_completed_total = Counter(
    'orders_completed_total',
    'Total completed orders',
    ['region', 'currency']
)

payment_failed_total = Counter(
    'payments_failed_total',
    'Total failed payment attempts',
    ['payment_method', 'failure_reason']
)

# In your signup handler
signup_total.labels(country='US', plan_type='pro').inc()

# In your payment processor
if payment_succeeded:
    order_completed_total.labels(region='EMEA', currency='EUR').inc()
else:
    payment_failed_total.labels(
        payment_method='card',
        failure_reason='insufficient_funds'
    ).inc()

These counters stream to Prometheus and let you query trends instantly. In Grafana, you can compute the rate of signups per minute using rate(signups_total[5m]), giving you real-time visibility into whether your marketing campaigns are working or if your signup flow is broken.

The beauty of this approach is that it integrates seamlessly with your existing monitoring infrastructure. Your on-call engineer can see business metrics and system metrics on the same dashboard, making root cause analysis faster. If orders drop 80% and you simultaneously see a spike in payment timeouts, the connection is immediate.

Avoid High-Cardinality Labels

This is where most teams stumble. Prometheus stores each unique combination of metric name plus label values as a separate time series. If you label orders_completed_total with user_id, and you have millions of users, you've created millions of time series. Prometheus will struggle, your storage will balloon, and queries will slow to a crawl.

High cardinality happens insidiously. Consider this problematic approach:

# AVOID THIS - millions of unique user_ids
order_total = Counter(
    'orders_total',
    'Orders by user',
    ['user_id', 'product_id', 'session_id']  # Cardinality explosion
)

# Each order creates a new time series
order_total.labels(user_id='usr_12345', product_id='prod_9876', session_id='sess_xyz').inc()

Instead, keep labels bounded to low-cardinality dimensions that actually matter for slicing:

# BETTER - bounded dimensions
order_total = Counter(
    'orders_total',
    'Orders by region and plan type',
    ['region', 'plan_tier', 'currency']
)

# Max ~5 regions × 3 tiers × 5 currencies = ~75 time series
order_total.labels(region='US', plan_tier='premium', currency='USD').inc()

A practical heuristic: if a label value could be in the hundreds or more, it's too high-cardinality. Think regions, tiers, payment methods—not user IDs, session IDs, or transaction IDs. For individual transaction details, write to your database. For aggregate trends, use bounded labels in Prometheus.

If you need to track which specific customer had a spike in orders (troubleshooting), query your database. If you need to see that "premium tier customers in APAC account for 35% of revenue this week," that's Prometheus's job with well-chosen labels.

Great for Anomaly Alerts

Business metrics shine when combined with anomaly detection. Unlike binary alerts ("orders per minute below 10"), anomaly detection learns normal patterns and fires when behavior deviates significantly. Your team ships a breaking change to checkout on Friday evening? An anomaly alert catches it before you're paged by customers.

Set up a baseline recording rule in Prometheus to smooth out daily and weekly patterns:

groups:
  - name: business_metrics
    rules:
      - record: orders:rate5m
        expr: rate(orders_completed_total[5m])
      
      - record: signups:rate5m
        expr: rate(signups_total[5m])
      
      - record: payment_fail_ratio:5m
        expr: |
          rate(payments_failed_total[5m])
          /
          (rate(payments_failed_total[5m]) + rate(payments_succeeded_total[5m]))

Then layer Prometheus's built-in predict_linear() function or integrate with an external anomaly engine like Moogsoft or custom code in your alerting layer:

alert: OrdersAnomalyDetected
expr: |
  abs(rate(orders_completed_total[5m]) - predict_linear(orders_completed_total[1h], 5*60))
  / predict_linear(orders_completed_total[1h], 5*60) > 0.3
for: 5m
annotations:
  summary: "Orders {{ $value | humanizePercentage }}% below predicted baseline"
  dashboard: "https://grafana.internal/d/business-metrics"

This alert triggers when actual order rate deviates more than 30% from the linear prediction over the last hour. In practice, you'd also exclude known maintenance windows and adjust thresholds per business cycle (holiday spikes, etc.). The key is that revenue-impacting problems surface automatically before manual investigation.

Keep Financial Truth in Your Database

Here's a critical boundary: Prometheus is not your source of truth for financial data. It's your early warning system. All actual transactions, revenue figures, and customer accounting must live in your transactional database with full ACID guarantees, audit trails, and compliance controls.

Think of Prometheus business metrics as a real-time proxy—they're accurate enough for alerting and dashboarding, but not for billing. Here's why:

• Prometheus data expires. Default retention is 15 days. If you need historical revenue reports for last quarter, you're reading from PostgreSQL, not Prometheus.
• Prometheus doesn't guarantee persistence. A disk failure or misconfigured retention could lose data. Your billing system can't afford that risk.
• Prometheus metrics are approximate. Cardinality pruning, sampling in heavy load, and timestamp rounding can introduce minor inaccuracies. Your accountant won't accept "approximately $2.5M in revenue."
• Compliance and audit trails. Financial regulations demand complete transaction logs with timestamps, user actions, and reversals. Prometheus isn't designed for that.

The healthy architecture: emit business events to both systems. Write to Kafka or a message queue, then fan out to Prometheus (via a collector) and your financial database:

# Pseudo-code flow
def complete_order(order):
    # Persist to authoritative DB first
    db.insert('orders', {
        'id': order.id,
        'user_id': order.user_id,
        'amount': order.amount,
        'currency': order.currency,
        'timestamp': now(),
        'status': 'completed'
    })
    
    # Emit event to Kafka
    kafka.produce('business-events', {
        'event_type': 'order.completed',
        'order_id': order.id,
        'amount': order.amount,
        'region': order.region,
        'plan_tier': order.plan_tier,
        'timestamp': now()
    })
    
    # Kafka consumer increments Prometheus counter
    # (async, fire-and-forget)
    metrics.orders_completed_total.labels(
        region=order.region,
        plan_tier=order.plan_tier
    ).inc()

This separation ensures your alerting and dashboards remain fast and responsive, while your financial records stay accurate and compliant.

Common Mistakes

• Labeling with timestamps or IDs. Don't include request IDs, trace IDs, or Unix timestamps as labels. Use a single label like outcome (success/failure) or error_type (if cardinality is low).
• Tracking individual user journeys in Prometheus. That's what your analytics database or event streaming platform is for. Prometheus is aggregate-only.
• Mixing financial and operational metrics on the same dashboard. Use separate dashboards with different refresh rates. Financial dashboards can be slower (updated every few minutes); operational dashboards should update every 10–30 seconds.
• Alert thresholds based on absolute numbers. "Alert if orders < 100" breaks when you scale. Use ratios, moving averages, or anomaly detection instead.
• Forgetting to instrument negative outcomes. Track signups_total but also signup_failures_total (by failure reason). You need the full picture to spot problems.

Key Takeaways

• Use Prometheus counters for monotonic business events: signups, orders, payments, cancellations, and refunds.
• Keep labels low-cardinality (regions, plan tiers, currency) and never use user IDs, session IDs, or timestamps as labels.
• Combine business metrics with anomaly detection to catch revenue-impacting bugs before customers do.
• Use Prometheus for real-time alerting and dashboarding, but always persist financial truth in your transactional database with full audit trails.
• Instrument both success and failure paths—understand why customers drop off, not just that they do.