Grafana alerting: Monitoring best practices explained

Grafana alerting: Monitoring best practices explained

After months of collecting metrics, logs, and traces across your infrastructure, the real value emerges when systems wake you up at 3 AM with actionable intelligence—not noise. Grafana alerting: Monitoring best practices explained is about centralizing alerts from any data source, routing them intelligently, and ensuring your team actually responds to what matters. In this article, we'll walk through building a reliable alerting system that serves Prometheus metrics, Loki logs, and database queries through a single, manageable interface.

Unified alerting across sources

One of Grafana's strongest features is the ability to define alerts once and evaluate them against multiple data sources without rewriting logic. Instead of maintaining separate alerting rules in Prometheus, Loki, and your monitoring tool of choice, you configure everything in Grafana's unified alerting engine.

This approach eliminates duplication. A high-memory alert doesn't need to exist in three places with three different thresholds. It lives in one place, evaluates on a schedule you control, and fires through channels you've explicitly chosen.

Consider a typical ecommerce platform. You might alert on:

• Prometheus metrics for request latency and error rates
• Loki logs for authentication failures or payment gateway timeouts
• PostgreSQL queries for inventory levels dropping below safety thresholds

Rather than juggling three separate alerting systems, Grafana's unified alerting evaluates all three as part of the same pipeline. You define conditions, grouping strategies, and notification behavior in one place.

Here's how you'd structure a simple alert rule that queries Prometheus:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-latency-alert
spec:
  groups:
  - name: api_alerts
    interval: 1m
    rules:
    - alert: HighAPILatency
      expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1
      for: 5m
      annotations:
        summary: "API p95 latency exceeds 1 second"
        description: "Current value: {{ $value }}s"

In Grafana's unified alerting, you'd create an alert rule via the UI or provisioning that references this metric. The key advantage: once configured, you manage notification routing centrally, regardless of source.

Contact points and notification policies

An alert that fires but never reaches anyone is worse than no alert. Grafana separates alert definition from alert delivery through contact points and notification policies.

A contact point is a destination: Slack channel, PagerDuty escalation, email inbox, or webhook endpoint. You define these once and reference them by name in policies.

A notification policy is a tree of rules that matches incoming alerts against labels and routes them to the right contact point. Think of it as your incident triage system.

Here's a practical example. Your team runs three services: payments, inventory, and auth. Payment failures are critical; inventory warnings are informational.

apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      resolve_timeout: 5m
    
    contact_points:
    - name: payments-pagerduty
      pagerduty_configs:
      - service_key: ${PAGERDUTY_KEY}
    
    - name: inventory-slack
      slack_configs:
      - api_url: ${SLACK_WEBHOOK}
          channel: '#inventory-alerts'
    
    - name: auth-email
      email_configs:
      - to: security-team@company.com
          from: alerts@company.com
    
    routing:
      receiver: 'default'
      group_by: ['alertname', 'cluster', 'service']
      group_wait: 10s
      group_interval: 10s
      repeat_interval: 12h
      
      routes:
      - match:
          service: payments
        receiver: payments-pagerduty
        continue: false
      
      - match:
          service: inventory
        receiver: inventory-slack
        severity: warning
      
      - match:
          service: auth
        receiver: auth-email

This configuration ensures that when a payment alert fires, it pages the on-call engineer via PagerDuty immediately. Inventory warnings go to Slack for visibility. Auth issues reach the security team's inbox. Each team gets exactly what they need to act.

In Grafana, you set this up by navigating to Alerting > Contact Points, creating each destination, then defining your routing policy tree. The UI makes it visual and prevents typos in YAML.

Alert on Prometheus, Loki and SQL

Grafana's query language flexibility lets you write alerts against any backend. This is where unified alerting proves its value in practice.

Prometheus alerts

expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05

This triggers when the error rate exceeds 5% over five minutes.

Loki alerts

expr: count_over_time({job="api"} |= "OutOfMemory" [5m]) > 10

This fires when more than 10 "OutOfMemory" messages appear in your API logs within five minutes. No intermediate metric scraper needed.

SQL alerts

SELECT COUNT(*) as expired_sessions
FROM user_sessions
WHERE expires_at < NOW()
AND NOT (marked_expired = true)

If this query returns more than a threshold of zombie sessions, it alerts your database team.

In Grafana, you create an alert rule, choose your data source (Prometheus, Loki, PostgreSQL), write your query, set a threshold condition, and define how long the condition must persist before firing. The evaluation engine runs on Grafana's schedule—usually every minute—and tracks state across evaluations.

Choose one alerting system per team

We've seen teams paralyzed by alert fatigue because three different systems fire the same alert with different thresholds, timing, and routing. A better approach: choose Grafana as your single source of truth for alerting logic across the entire organization.

This doesn't mean ripping out Prometheus alerting rules tomorrow. It means new alerts go into Grafana. Existing Prometheus rules can coexist—Grafana can even display and manage them—but avoid the chaos of alerts originating from multiple systems.

Why a single system?

• Consistency: All teams understand the same alert definition format and routing language.
• Maintenance: When thresholds change, you update one place, not three.
• Tracing: When an alert doesn't fire as expected, debugging is straightforward—you know exactly where the rule lives and which contact points it routes through.
• On-call visibility: Your on-call engineer sees all alerts in one interface, not scattered across Prometheus, AlertManager, and Datadog.

In organizations running both Prometheus and Grafana, the pattern is clear: Prometheus scrapes metrics and evaluates simple threshold rules. Grafana takes ownership of complex multi-source logic, routing decisions, and suppression policies. When you need to correlate a metric spike with a log anomaly and a database query result, Grafana becomes indispensable.

Common mistakes

Setting alerts too sensitive: An alert firing every 30 seconds trains your team to ignore all alerts. Start conservative. A 5-minute evaluation window catches real problems without noise.

Forgetting to test routing: You discover your alert route is broken only when it matters most. Test each contact point by triggering a test alert. Verify Slack channels exist, PagerDuty keys are valid, and email deliverability is working.

Mixing data sources without context: An alert that triggers on both a Prometheus metric and a Loki log pattern might make sense, but without clear annotations explaining which condition fired, your on-call engineer wastes time in the dashboard. Always include a summary and runbook link.

Neglecting alert fatigue reviews: Set a calendar reminder monthly to review which alerts actually led to meaningful action and which ones fired but resolved on their own. Silence or remove the noise.

Hardcoding thresholds: Use variables and template variables in Grafana alert rules so you can adjust thresholds without editing the rule itself. A 95th percentile latency threshold might differ between your API gateway and your background job service.

Key takeaways

• Grafana's unified alerting centralizes alert definition and routing, eliminating duplication across Prometheus, Loki, and SQL sources.
• Separate concerns: define alert conditions in alert rules, then route alerts to contact points (Slack, PagerDuty, email) via notification policies that match on alert labels.
• Alert on metrics, logs, and database queries without maintaining three separate alerting systems—write queries in your data source's native language (PromQL, LogQL, SQL) and let Grafana orchestrate evaluation.
• Establish one alerting system as your team's source of truth to reduce noise, improve consistency, and simplify debugging when alerts misbehave.
• Test alert routing, keep evaluation windows reasonable to avoid false positives, and regularly review which alerts drive real action.