Monitoring and alarms with Terraform in Terraform & IaC

Introduction

On day 67 of building infrastructure with Terraform, we move past provisioning and into the reality every team faces in production: something breaks at 3 AM, and your team needs to know about it instantly. Monitoring and alarms with Terraform in Terraform & IaC means treating your observability stack exactly like your infrastructure—defined in code, versioned, tested, and replicated consistently across dev, staging, and production. Instead of clicking through the AWS console to configure CloudWatch alarms for each environment separately, we define them once in HCL and deploy them everywhere.

This approach eliminates drift, reduces human error, and ensures that the alerting thresholds you carefully tuned in production are the same ones protecting your staging environment. Let's walk through how to make observability part of your infrastructure-as-code practice.

CloudWatch Alarms for Errors and Latency

CloudWatch alarms are the foundation of AWS observability. They watch metrics over time and trigger actions when thresholds are breached. The key to managing them in Terraform is understanding that each alarm targets a specific metric, dimension set, and statistic. We define these once, and they propagate to every environment through variables.

Consider a real scenario: you operate an API endpoint that normally responds in under 200ms. You want to be alerted when the p99 latency exceeds 1 second for five consecutive minutes. Here's how you'd define that alarm in Terraform:

resource "aws_cloudwatch_metric_alarm" "api_latency_high" {
  alarm_name          = "api-p99-latency-high-${var.environment}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Duration"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "p99"
  threshold           = 1000
  alarm_description   = "Alert when API p99 latency exceeds 1 second"
  treat_missing_data  = "notBreaching"

  dimensions = {
    FunctionName = aws_lambda_function.api.function_name
  }

  alarm_actions = [aws_sns_topic.alerts.arn]
}

The treat_missing_data parameter deserves attention. Set it to notBreaching unless you want false positives when a function stops receiving traffic. The evaluation_periods and period combination creates a sliding window—in this case, we check the p99 every 5 minutes and trigger only if the threshold is breached in at least 1 consecutive period.

For application errors, you'd want a similar alarm but targeting error counts:

resource "aws_cloudwatch_metric_alarm" "lambda_errors" {
  alarm_name          = "lambda-errors-${var.environment}"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 60
  statistic           = "Sum"
  threshold           = 5
  alarm_description   = "Alert when Lambda errors exceed 5 in 1 minute"

  dimensions = {
    FunctionName = aws_lambda_function.api.function_name
  }

  alarm_actions = [aws_sns_topic.alerts.arn]
}

The difference here: we use Sum as the statistic because we want the total count of errors in a period, and we trigger immediately (1 evaluation period) because errors are urgent. In production, you might set a higher threshold; in staging, you might lower it to catch more issues before they reach production.

SNS Topics for Notifications

An alarm is only useful if it triggers an action. SNS (Simple Notification Service) topics are the delivery mechanism. You define a topic once, subscribe the endpoints you need (email, Slack, PagerDuty, Lambda), and every alarm in your infrastructure can publish to it.

Here's the pattern we use across our teams:

resource "aws_sns_topic" "alerts" {
  name = "infrastructure-alerts-${var.environment}"

  tags = {
    Environment = var.environment
    ManagedBy   = "Terraform"
  }
}

resource "aws_sns_topic_subscription" "alerts_email" {
  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "email"
  endpoint  = var.alert_email

  depends_on = [aws_sns_topic.alerts]
}

# For production, you might also add a PagerDuty subscription
resource "aws_sns_topic_subscription" "alerts_pagerduty" {
  count = var.environment == "production" ? 1 : 0

  topic_arn = aws_sns_topic.alerts.arn
  protocol  = "https"
  endpoint  = var.pagerduty_integration_url

  depends_on = [aws_sns_topic.alerts]
}

Notice the conditional on the PagerDuty subscription. In staging, you don't need to wake someone up; in production, you do. Terraform variables let you express that logic declaratively. We store sensitive endpoints like PagerDuty URLs in secrets or environment variables, never in code.

When you terraform apply this, SNS will send a confirmation email to your address. You must click the link to confirm the subscription. In automation pipelines, we often use SNS Lambda subscriptions or Slack integrations instead of email to reduce manual confirmation steps.

Dashboards as Code

CloudWatch dashboards are where you visualize the health of your infrastructure. The AWS console offers a point-and-click interface, but that dashboard definition never makes it into version control. Terraform lets you define dashboards in code, ensuring consistency and allowing you to review changes in pull requests.

Dashboards are defined as JSON in Terraform. Here's a practical example for a microservice:

resource "aws_cloudwatch_dashboard" "api_health" {
  dashboard_name = "api-health-${var.environment}"

  dashboard_body = jsonencode({
    widgets = [
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/Lambda", "Duration", { stat = "p99" }],
            ["AWS/Lambda", "Errors", { stat = "Sum" }],
            ["AWS/Lambda", "Throttles", { stat = "Sum" }],
          ]
          period = 300
          stat   = "Average"
          region = var.aws_region
          title  = "Lambda Performance"
        }
      },
      {
        type = "metric"
        properties = {
          metrics = [
            ["AWS/RDS", "CPUUtilization"],
            ["AWS/RDS", "DatabaseConnections"],
          ]
          period = 60
          stat   = "Average"
          region = var.aws_region
          title  = "Database Health"
        }
      },
      {
        type = "log"
        properties = {
          query = "fields @timestamp, @message | filter @message like /ERROR/ | stats count() as error_count by bin(5m)"
          title = "Application Errors (5m bins)"
        }
      }
    ]
  })
}

The jsonencode function converts Terraform objects into valid JSON for CloudWatch. This approach is verbose but powerful: you can loop over services, inject environment-specific thresholds, and validate the structure in code rather than discovering mistakes in the console.

Many teams keep a separate dashboards.tf file so that dashboard changes are easy to spot in pull requests. A dashboard update that shifts a critical metric to a different panel should trigger a review.

Same Alarms in Every Environment

The real power of managing alarms as code emerges when you deploy to multiple environments. Using Terraform modules and variables, you define your alarms once and instantiate them everywhere.

Here's a module structure that works well in practice:

module "api_monitoring" {
  source = "./modules/api-monitoring"

  environment             = var.environment
  function_name           = aws_lambda_function.api.function_name
  sns_topic_arn           = aws_sns_topic.alerts.arn
  latency_threshold_ms    = var.environment == "production" ? 1000 : 5000
  error_threshold_count   = var.environment == "production" ? 5 : 50
  aws_region              = var.aws_region

  tags = local.common_tags
}

In the module (modules/api-monitoring/main.tf), you define the alarms once:

variable "function_name" { type = string }
variable "sns_topic_arn" { type = string }
variable "latency_threshold_ms" { type = number }
variable "error_threshold_count" { type = number }
variable "environment" { type = string }

resource "aws_cloudwatch_metric_alarm" "latency" {
  alarm_name          = "${var.function_name}-latency-${var.environment}"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 1
  metric_name         = "Duration"
  namespace           = "AWS/Lambda"
  period              = 300
  statistic           = "p99"
  threshold           = var.latency_threshold_ms
  alarm_actions       = [var.sns_topic_arn]

  dimensions = {
    FunctionName = var.function_name
  }
}

resource "aws_cloudwatch_metric_alarm" "errors" {
  alarm_name          = "${var.function_name}-errors-${var.environment}"
  comparison_operator = "GreaterThanOrEqualToThreshold"
  evaluation_periods  = 1
  metric_name         = "Errors"
  namespace           = "AWS/Lambda"
  period              = 60
  statistic           = "Sum"
  threshold           = var.error_threshold_count
  alarm_actions       = [var.sns_topic_arn]

  dimensions = {
    FunctionName = var.function_name
  }
}

Now you call this module for dev, staging, and production, passing different thresholds. Your staging and production alarms have the same structure but different sensitivity. When you discover a threshold needs to change, you update one module and apply it across all environments in a single terraform plan.

This consistency prevents the classic scenario where production alerts are stricter than staging, so you only discover problems in prod. It also makes onboarding new services predictable: copy the module call, adjust the thresholds for your risk tolerance, and you have full observability immediately.

Common Mistakes

• Not setting treat_missing_data. By default, CloudWatch alarms enter an INSUFFICIENT_DATA state when there's no data. This can cause false positives when a function receives no traffic. Set it to notBreaching for graceful handling of quiet periods.
• Choosing the wrong statistic. Average can mask spikes. For latency, use p99 or p95. For errors, use Sum in a short period (60 seconds) so you catch issues before they accumulate. Understand the difference between Average, Maximum, and percentile statistics.
• Hardcoding threshold values in Terraform. Thresholds drift. Put them in terraform.tfvars or environment-specific files so you can change them without editing the module. This also makes it easy to experiment with stricter thresholds in dev before applying them to production.
• Creating alarms without notification channels. An alarm that doesn't notify anyone is useless. Always pair alarms with SNS topics and subscriptions. Test the chain end-to-end: trigger the alarm manually and verify you receive the notification.
• Forgetting to destroy test alarms. When experimenting, you might create a dozen test alarms. Use terraform destroy to clean them up, or they'll clutter your AWS account and your dashboards.
• Different alarm configurations per environment. If your prod alarms have a 5-minute evaluation window and your staging alarms have 1 minute, you'll tune differently and make mistakes. Use modules to enforce consistency; only vary thresholds, not structure.

Key Takeaways

• Define CloudWatch alarms in Terraform using aws_cloudwatch_metric_alarm resources. Use appropriate statistics (p99 for latency, Sum for error counts) and set treat_missing_data to prevent false positives.
• Pair every alarm with an SNS topic so alerts reach your team. Use conditional subscriptions (e.g., PagerDuty only in production) to route urgency appropriately.
• Define dashboards as code using jsonencode and aws_cloudwatch_dashboard. Review dashboard changes in pull requests just like infrastructure changes.
• Use Terraform modules to define alarm templates once, then instantiate them across dev, staging, and production with environment-specific thresholds. This ensures consistency while allowing tuning per environment.
• Store threshold values in variables or tfvars files, never hardcoded in modules. This makes it easy to experiment and adjust without re-editing code.
• Test your alerting chain end-to-end: verify that an alarm actually triggers and that you receive the notification through your chosen channel.
• Version control your monitoring configuration just like your infrastructure. Alert rule changes should go through code review, and you should be able to roll back a bad threshold change with a single terraform apply.