Silences and maintenance windows: lessons from production

Silences and maintenance windows: lessons from production

You're pushing a critical database migration to production at 2 AM, and within seconds your on-call engineer's phone lights up with hundreds of alerts about connection timeouts and slow queries. The migration is proceeding exactly as planned, but your monitoring system is screaming anyway. This is where silences and maintenance windows become essential—they let you quiet alerts during planned work without disabling monitoring altogether.

In this article, we'll walk through Alertmanager's silencing features, show you how to match alerts by their labels, set proper expiry times, and keep your team in sync. These practices emerge from hard-won production experience where the difference between a smooth maintenance window and a chaotic one often comes down to how well you've managed your alerts.

Why silences matter in production

Silencing is not about hiding problems—it's about trust and signal-to-noise ratio. When your team knows that a spike in error rates is expected because you're deploying a new service, they won't panic. When they receive only the alerts that matter right now, they can act decisively. Without silences, alert fatigue sets in fast. Your on-call team stops trusting the system, and genuine incidents get lost in the noise.

The alternative—disabling alerting rules or stopping Prometheus scraping—is dangerous. You lose visibility entirely. Silences give you a middle ground: targeted, time-bounded suppression of specific alert conditions.

Create silences in Alertmanager

Alertmanager exposes a simple HTTP API for creating and managing silences. The most straightforward way is through the Alertmanager UI, but production teams typically automate this via API calls from their deployment pipelines.

Here's what a silence request looks like:

POST /api/v1/silences HTTP/1.1
Content-Type: application/json

{
  "matchers": [
    {
      "name": "alertname",
      "value": "HighErrorRate",
      "isRegex": false
    }
  ],
  "startsAt": "2024-01-17T02:00:00.000Z",
  "endsAt": "2024-01-17T03:00:00.000Z",
  "createdBy": "deployment-pipeline",
  "comment": "Database migration: expected connection pool exhaustion during cutover"
}

The response includes a silence ID that you can use to track or delete the silence later:

{
  "silenceID": "a1b2c3d4e5f6"
}

In practice, we recommend wrapping this in a helper script or function that your team runs before maintenance begins. Here's a simple curl-based example:

#!/bin/bash
ALERTMANAGER_URL="http://alertmanager.internal:9093"
ALERT_NAME="HighErrorRate"
DURATION_MINUTES=60
REASON="$1"

START_TIME=$(date -u +"%Y-%m-%dT%H:%M:%S.000Z")
END_TIME=$(date -u -d "+${DURATION_MINUTES} minutes" +"%Y-%m-%dT%H:%M:%S.000Z")

curl -X POST "${ALERTMANAGER_URL}/api/v1/silences" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "matchers": [
    {
      "name": "alertname",
      "value": "${ALERT_NAME}",
      "isRegex": false
    }
  ],
  "startsAt": "${START_TIME}",
  "endsAt": "${END_TIME}",
  "createdBy": "automation",
  "comment": "${REASON}"
}
EOF

Run this before your maintenance window starts, and Alertmanager will stop routing alerts matching your criteria to any notification channel.

Match by labels

The power of silences lies in label matching. Alertmanager evaluates matchers against alert labels—both those generated by Prometheus rules and those added by your scrape configs. You can silence based on alert name, service, environment, severity, or any custom label you've defined.

Let's say you have several microservices and you're only deploying the payment service. You could silence only the alerts related to that service:

{
  "matchers": [
    {
      "name": "service",
      "value": "payment",
      "isRegex": false
    }
  ],
  "startsAt": "2024-01-17T02:00:00.000Z",
  "endsAt": "2024-01-17T03:00:00.000Z",
  "createdBy": "deployment",
  "comment": "Payment service blue-green deployment"
}

Or use regular expressions to match multiple related alerts:

{
  "matchers": [
    {
      "name": "alertname",
      "value": "High.*ErrorRate",
      "isRegex": true
    },
    {
      "name": "environment",
      "value": "staging",
      "isRegex": false
    }
  ],
  "startsAt": "2024-01-17T02:00:00.000Z",
  "endsAt": "2024-01-17T02:30:00.000Z",
  "createdBy": "qa-team",
  "comment": "Staging load test"
}

Notice we're combining matchers: both conditions must match for an alert to be silenced. This precision prevents accidentally silencing unrelated alerts. We've learned the hard way that overly broad silences lead to missing real problems. Always be as specific as your labels allow.

Set an expiry

Never create an open-ended silence. Always set an explicit end time. This is a safety mechanism: even if someone forgets to remove the silence, it expires automatically.

A good rule of thumb is to set the expiry slightly longer than you expect the maintenance to take. If you're migrating a database and expect 45 minutes, set the silence for 90 minutes. This gives you buffer time without creating indefinite dark periods in your monitoring.

Alertmanager's UI shows all active silences with their remaining time. Make it part of your pre-maintenance checklist to verify the silence is in place:

If your maintenance finishes early, delete the silence immediately. Use the silence ID returned when you created it:

DELETE /api/v1/silences/a1b2c3d4e5f6

Or via curl:

curl -X DELETE http://alertmanager.internal:9093/api/v1/silences/a1b2c3d4e5f6

Deleting early signals to your team that you're back to normal and monitoring is fully active again.

Announce maintenance to the team

The technical silence is only half the story. The other half is communication. Before you create a silence, post a message in your team's incident channel or Slack with the following details:

• What: Brief description of the work (e.g., "Database schema migration")
• When: Exact start and end times in UTC
• Who: The engineer or team responsible
• Expected impact: Which services or features may be affected
• Escalation: A phone number or Slack handle for urgent issues

Here's a template we use in production:

🔧 Maintenance Window: Database Connection Pool Upgrade

Start: 2024-01-17 02:00 UTC | End: 2024-01-17 03:30 UTC | Duration: ~90 min

Owner: @alice (on-call backup: @bob)

Expected impact: Transient 503 errors on checkout, 2–5 sec latency spikes

Silences active for: HighErrorRate, HighLatency (checkout service)

Real emergencies: page Alice at +1 555-0123 or #incident Slack

This message sets expectations and ensures that if something genuinely unexpected happens—not the routine timeouts you're expecting, but a cascade failure—your team knows they should still escalate.

Common mistakes

Creating silences too early or too late. If you silence alerts 2 hours before maintenance, you might miss early warning signs that something's already breaking. Create silences just before the window begins. Similarly, deleting silences while work is still in progress leaves you exposed to false alerts.

Silences that are too broad. A silence matching only environment=production is dangerous—it could hide real issues in other services. Always combine matchers to be as specific as possible. Use service name, alert name, or both.

Forgetting to communicate. A silence without a message in your incident channel leads to confusion: Why are these alerts missing? Is the system broken? Make communication automatic: a Slack bot that posts whenever a silence is created helps a lot.

Not setting an expiry. We've seen silences accidentally left in place for days. Always set explicit end times, no exceptions.

Silencing in the wrong Alertmanager. If you're running multiple Alertmanager instances (one per environment, for example), remember that silences don't replicate. You might silence staging but forget production. Double-check your API endpoint.

Key takeaways

• Silences suppress alerts during planned maintenance without disabling monitoring rules or scraping.
• Use the Alertmanager HTTP API or UI to create silences with label matchers, start/end times, and a comment explaining why.
• Match alerts by label as precisely as possible—combine multiple matchers to avoid over-silencing.
• Always set an explicit expiry time; never create open-ended silences.
• Announce the maintenance window to your team before creating silences, explaining what's happening and how to escalate real emergencies.
• Delete silences as soon as maintenance finishes to resume full monitoring.
• Make silence creation and deletion part of your deployment pipeline to reduce manual steps and human error.