Kibana alerting: Elasticsearch best practices explained

Kibana alerting: Elasticsearch best practices explained

Most teams discover alerting too late—after a silent outage has already cost them customers. Kibana alerting: Elasticsearch best practices explained gives you the foundation to detect problems before they become incidents. In this article, we'll walk through building reliable alert rules, configuring connectors, and eliminating the noise that makes engineers ignore notifications.

Rules on queries and thresholds

A Kibana alert rule sits on top of a saved query. The rule evaluates that query on a schedule, compares the result against a threshold, and fires when conditions are met. The quality of your rule depends entirely on the precision of your query and the sensibility of your threshold.

Let's say you're monitoring request latency. A naive rule might alert when http.response_time_ms > 5000, but that fires constantly during normal traffic peaks. Instead, you want to alert when latency is anomalous—when it's significantly higher than baseline for your time of day.

Build your query first in Kibana's Discover view. Use aggregations to surface patterns:

{
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-15m" } } },
        { "term": { "service.name": "checkout-api" } },
        { "range": { "http.response_time_ms": { "gte": 3000 } } }
      ]
    }
  },
  "aggs": {
    "error_rate": {
      "avg": { "field": "http.response_time_ms" }
    }
  }
}

Once your query is saved, create a rule from it. Set the rule to run every 5 minutes and check the last 15 minutes of data—this window gives you enough signal without excessive lag. For the threshold, use the 95th percentile of your baseline metric rather than a fixed number. Most alerting rules allow comparisons like "is above" or "is below," and many support dynamic thresholds calculated from historical data.

In practice, you'll often need multiple rules per metric. One rule might alert on absolute values (response time > 10 seconds always matters), while another alerts on percent-change (a 50% jump in error rate, even if still low in absolute terms). Stack these rules to catch different failure modes.

Connectors for Slack and email

Kibana connectors deliver alert notifications to your team. The two most practical connectors for development teams are Slack and email, though PagerDuty, Opsgenie, and webhook connectors are also common.

For Slack, create a connector in Kibana Stack Management:

1. Navigate to Stack Management > Connectors
2. Click Create connector and choose Slack
3. Paste your webhook URL from your Slack workspace
4. Test the connection and save

When you attach the Slack connector to a rule, you can customize the message. Use templating variables to include relevant context:

{
  "message": "Alert: {{rule.name}} triggered at {{date date 'yyyy-MM-dd HH:mm:ss'}}\nService: {{context.service}}\nValue: {{context.value}}\nThreshold: {{context.threshold}}\nLink: {{context.link}}"
}

Email connectors work similarly but suit digest-style alerts—when you want a summary sent daily rather than per-incident. Use email for less-urgent observability metrics and Slack for true incidents that need immediate attention.

The key difference: Slack goes to a channel where everyone sees it. Email goes to an inbox where it's easy to miss. Use Slack for SEV-1 incidents, email for trend reports.

Error spike alerts

One of the most impactful alerts you can build detects error spikes—sudden increases in failure rate. This catches bad deployments, downstream service issues, and resource exhaustion before users complain.

Build the query by filtering for errors and comparing current rate to a baseline:

{
  "query": {
    "bool": {
      "filter": [
        { "range": { "@timestamp": { "gte": "now-5m" } } },
        { "term": { "log.level": "error" } },
        { "term": { "service.name": "payment-processor" } }
      ]
    }
  },
  "aggs": {
    "error_count": {
      "value_count": { "field": "event.id" }
    }
  }
}

Set the threshold relative to your baseline. For example: "alert if error count in the last 5 minutes is more than 2x the average of the last hour." This catches sudden spikes while ignoring the seasonal variation in your error baseline.

The rule should run frequently—every 1–2 minutes for critical services. When it fires, include the top error messages and affected endpoints in the notification. This saves your on-call engineer from hunting through logs:

{
  "message": "Error spike detected in payment-processor\nRate: {{context.current_rate}} errors/min (baseline: {{context.baseline_rate}})\nTop error: {{context.top_error_message}}\nAffected endpoint: {{context.top_endpoint}}"
}

Combine your error spike alert with a log drill-down link in the Slack message. Include a URL that opens Kibana's Discover view filtered to the error logs from the last 5 minutes. Engineers can then investigate without context-switching:

https://kibana.example.com/app/discover#/?_a=(query:(match:(log.level:(query:'error'))))

Avoid noisy rules

Most alerting systems fail not because they miss incidents, but because they fire constantly. Noisy rules train engineers to ignore all alerts. You'll then miss the critical ones.

Common sources of noise:

• Thresholds set too low. If your CPU alert fires when CPU hits 70%, it fires 20 times a day during normal operation.
• Rules that don't account for time-of-day or day-of-week patterns. Traffic at midnight is different from peak hours.
• Rules with no aggregation window. A single slow request shouldn't trigger an alert if thousands more completed normally.
• Flapping rules. A rule that fires and resolves every 2 minutes creates message spam.

To eliminate noise, use these strategies:

Add a de-duplication window. Configure the rule to only send a notification once per hour, even if it remains in an alert state. In Kibana, set Throttle to your desired interval (e.g., 1 hour). This prevents one incident from generating dozens of Slack messages.

Use percentiles instead of raw thresholds. Instead of "alert if CPU > 75%", use "alert if CPU > 95th percentile of last 7 days". This adapts to your actual baseline without manual tuning.

Add condition grouping. Require multiple conditions to be true before firing. For example, alert on error spike only if it's also accompanied by an increase in latency. This filters out transient blips.

Implement recovery actions. Set the rule to automatically resolve after the condition is no longer met, and send a recovery notification. Engineers want to know when the problem is fixed, not just when it starts.

Test your rules in a staging environment for at least a week before enabling them in production. Run the query manually at different times of day. If you see it would have fired during normal operation, adjust the threshold.

Common mistakes

• Alerting on raw counts instead of rates. An application that logs 1,000 errors during 100,000 requests (1% error rate) is healthier than one that logs 10 errors during 100 requests (10% error rate). Use percentage or ratio-based metrics.
• Ignoring timezone differences. If your team spans timezones, your "business hours" alert might fire outside business hours for someone. Use UTC for all time-based thresholds.
• Creating one massive rule instead of focused ones. A rule that checks 10 different conditions becomes difficult to debug. Create separate rules for separate concerns. You can group them visually in Kibana later.
• Not testing alert actions. Your connector might be misconfigured, your webhook might be invalid, or your Slack token might have expired. Test every alert action before relying on it.
• Setting the evaluation interval too short. Running a complex query every 10 seconds hammers Elasticsearch. Most practical alerts run every 5 minutes. Match your interval to how quickly you need to respond.

Key takeaways

• Build alert rules on top of well-tested queries, and always validate your query in Discover before creating the rule.
• Use dynamic thresholds (percentiles, rate-of-change) instead of fixed values to reduce false positives.
• Configure Slack for immediate incidents and email for trend reports. Always include drill-down links in notifications.
• Error spike alerts are high-impact: catch sudden increases in failure rate by comparing current rate to a historical baseline.
• Throttle repeated notifications to prevent alert fatigue. De-duplication windows of 1 hour are common for non-critical alerts.
• Test rules in staging and review them after a week in production. Adjust thresholds based on what you actually observe, not theory.
• Simple, focused rules are better than complex ones. Stack multiple single-purpose rules to catch different failure modes.