Blackbox exporter for uptime checks: a step-by-step guide

Introduction

Prometheus excels at scraping metrics from services you own, but what happens when you need to monitor endpoints from the outside looking in? That's where the blackbox exporter comes in. This guide covers blackbox exporter for uptime checks: a step-by-step guide to probing endpoints from external vantage points, catching real-world failures that internal metrics might miss.

Most teams already have alerting on response times and error rates pulled directly from application logs. What they're missing is the external user's perspective—whether your API is reachable, whether your TLS certificate is about to expire, or whether DNS resolution works from a customer's location. The blackbox exporter solves this by running synthetic checks against your services and exposing the results as Prometheus metrics.

Modules — HTTP, TCP, ICMP and DNS probes

The blackbox exporter ships with four main probe types, each suited to different scenarios. Understanding when to use each module is the foundation of effective external monitoring.

HTTP module is the workhorse. It sends HTTP requests (GET, POST, HEAD) to your endpoints and records status codes, response times, and SSL certificate metadata. Use this for web APIs, health checks, and content verification. TCP module simply opens a connection and checks if the port is listening—no application-level protocol needed. This works for databases, cache servers, and proprietary services. ICMP module sends ping packets to test network connectivity and latency without layer 7 complexity. DNS module queries nameservers directly, verifying that your domains resolve correctly.

Here's a minimal blackbox configuration with all four modules:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: "ip4"
      method: GET
  
  tcp_connect:
    prober: tcp
    timeout: 5s
    tcp:
      ip_protocol_fallback: true
  
  icmp_ping:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"
  
  dns_lookup:
    prober: dns
    timeout: 5s
    dns:
      preferred_ip_protocol: "ip4"
      nameserver: "8.8.8.8"

Each module has sensible defaults. The timeout field prevents probes from hanging indefinitely. The preferred_ip_protocol defaults to "ip4" but you can test IPv6 by setting it to "ip6". For production, you'll typically have multiple modules—HTTP probes for your public APIs, TCP probes for your database replicas, and DNS probes for critical domain names.

Probe URLs — Check status codes and response content

A probe URL tells the blackbox exporter where to look and what to expect. Prometheus scrapes the blackbox exporter with a target parameter containing the endpoint you want to check, and an optional module parameter specifying which probe module to use.

Consider a team running an e-commerce platform. You want to probe your main API at https://api.example.com/health, your checkout endpoint at https://api.example.com/v1/checkout, and your product search at https://api.example.com/search. Rather than duplicating targets, you configure Prometheus to hit the blackbox exporter with those URLs as parameters.

Here's how Prometheus scrape config looks:

scrape_configs:
  - job_name: 'blackbox'
    metrics_path: /probe
    static_configs:
      - targets:
        - https://api.example.com/health
        - https://api.example.com/v1/checkout
        - https://api.example.com/search
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115

The relabel_configs are crucial: the first extracts the target URL, the second preserves it as the instance label for clarity in alerts, and the third points to where your blackbox exporter is actually running (usually localhost:9115).

The blackbox exporter returns metrics like probe_success (1 if the probe succeeded, 0 if it failed), probe_duration_seconds (how long the probe took), and probe_http_status_code (the HTTP status returned). You can also validate response content with regex patterns:

modules:
  http_content_check:
    prober: http
    timeout: 5s
    http:
      method: GET
      body_size_limit: 10KB
      fail_if_body_matches_regexp:
        - "error"
        - "timeout"
      fail_if_not_matches_regexp:
        - "success"

This configuration checks that the response body contains "success" and doesn't contain "error" or "timeout". It's useful for verifying that an endpoint returns meaningful content, not just an HTTP 200 with an error message inside.

TLS expiry — probe_ssl_earliest_cert_expiry for certificate alerts

Certificate expiry is a common cause of production incidents. Unlike business logic bugs, expired certificates are deterministic and entirely preventable, yet many teams only discover the problem when customers start reporting HTTPS errors.

The blackbox exporter's HTTP module automatically extracts certificate metadata and exposes probe_ssl_earliest_cert_expiry, a Unix timestamp of when your certificate expires. Pair this with a simple Prometheus alert and you have early warning.

Set up a recording rule to convert the Unix timestamp into days remaining:

groups:
  - name: blackbox
    interval: 30s
    rules:
      - record: certificate:days_until_expiry
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400

Then alert when days remaining drops below 30:

      - alert: CertificateExpiringIn30Days
        expr: certificate:days_until_expiry < 30
        for: 1h
        annotations:
          summary: "Certificate expiring in {{ $value | humanize }} days"

The for: 1h prevents alerts from firing on transient issues. A certificate expiry metric won't flap—it's always true until renewal—so the alert fires immediately once the condition is met and stays firing until remediated.

When testing this locally, create a self-signed certificate with a short lifetime to verify the metric works:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 1 -nodes

Probe it with your blackbox exporter and check that probe_ssl_earliest_cert_expiry appears in the output. You should see a timestamp roughly 24 hours from now.

Multiple locations — Run probes from several regions

A single blackbox exporter checking your endpoint from one location gives you one perspective. If your service goes down globally, a single prober will catch it. But if your service is only broken in one AWS region, or from a specific ISP, a single prober might miss it.

The solution is running multiple blackbox exporters from different geographic locations. A team operating a travel platform might run probes from us-east, eu-west, and ap-southeast, ensuring that regional failures surface immediately.

Deploy blackbox exporter instances to each location:

---
# Deployment in us-east
apiVersion: apps/v1
kind: Deployment
metadata:
  name: blackbox-exporter-us-east
spec:
  replicas: 1
  selector:
    matchLabels:
      app: blackbox-exporter
      region: us-east
  template:
    metadata:
      labels:
        app: blackbox-exporter
        region: us-east
    spec:
      containers:
      - name: blackbox
        image: prom/blackbox-exporter:v0.24.0
        ports:
        - containerPort: 9115
        volumeMounts:
        - name: config
          mountPath: /etc/blackbox
        args:
        - --config.file=/etc/blackbox/blackbox.yml
      volumes:
      - name: config
        configMap:
          name: blackbox-config

In your Prometheus scrape config, differentiate probers by region:

scrape_configs:
  - job_name: 'blackbox-us-east'
    metrics_path: /probe
    static_configs:
      - targets:
        - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-us-east:9115
      - target_label: region
        replacement: us-east
  
  - job_name: 'blackbox-eu-west'
    metrics_path: /probe
    static_configs:
      - targets:
        - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-eu-west:9115
      - target_label: region
        replacement: eu-west

Now your Prometheus stores separate metrics for each region. A query like probe_success{region="us-east"} shows you only the us-east perspective. Create alerts that require all regions to fail before paging on-call, or alerts that fire as soon as any single region fails. This flexibility catches both global and regional issues.

Common mistakes

• Not setting sufficiently short scrape intervals. Blackbox exporter probes are cheap compared to running the actual service. Scraping every 60 seconds will miss brief outages. Use 15–30 second intervals for critical paths.
• Forgetting to set the module parameter. If you don't specify module: http_2xx in your scrape config, the blackbox exporter defaults to "http_2xx" but it's easy to accidentally omit it and end up probing the wrong module.
• Using HTTP probes for non-HTTP services. TCP checks are lighter and more appropriate for databases. An HTTP probe against a database port will timeout rather than cleanly fail.
• Not accounting for network latency in alert thresholds. A probe from a distant region legitimately takes longer. Set region-specific alert thresholds or use quantile-based rules.
• Scraping too many targets from a single exporter. The blackbox exporter is single-threaded by default. Probing hundreds of targets from one instance causes cascading timeouts. Scale horizontally when you exceed 50–100 targets per instance.
• Ignoring firewall rules. Outbound ICMP and DNS queries are often blocked. Test your probes from the actual network location before deploying.

Key takeaways

• The blackbox exporter provides HTTP, TCP, ICMP, and DNS probe modules. Pick the right one for your use case rather than defaulting to HTTP for everything.
• Use regex matching in HTTP probes to verify not just status codes but response content, catching silent failures that return 200 OK with an error message.
• Extract probe_ssl_earliest_cert_expiry and alert when certificates expire within 30 days. Certificate expiry is predictable and entirely preventable.
• Deploy multiple blackbox exporters across regions to catch both global and regional failures. Region-aware metrics enable sophisticated alerting that distinguishes between full outages and localized issues.
• Keep scrape intervals short (15–30 seconds), set reasonable timeouts, and scale horizontally as the number of targets grows.