CloudWatch log groups with Terraform in Terraform & IaC

Introduction

By day 90 of your infrastructure-as-code journey, you've likely deployed applications, databases, and networking across AWS. But logs—the often-forgotten pillar—still sprawl across CloudWatch without structure, retention policy, or cost governance. CloudWatch log groups with Terraform in Terraform & IaC is how we solve this at scale: declaring log infrastructure explicitly, controlling retention, encrypting sensitive data, and routing logs where they belong. This article shows you the production patterns we use across trading platforms, e-commerce services, and ride-hailing backends.

Why Manage Log Groups with Terraform

Many engineers assume AWS services create log groups on demand. They do—but leaving that to chance costs money and creates security gaps. When Lambda, ECS, or RDS auto-creates a log group, it inherits default settings: no retention limit (data lives forever, billing forever), no encryption, and no audit trail of who provisioned it.

Managing log groups explicitly in Terraform means:

• You own the retention policy from day one, capping storage costs.
• Encryption is baked in; no log group exists unencrypted.
• Every log group is versioned and reviewed in Git, creating accountability.
• You prevent the implicit creation of hundreds of orphaned log groups.

This is especially critical in regulated environments—financial services, healthcare, travel—where logs are audit evidence and must be managed with the same rigor as any other data resource.

Create Log Groups Before Services Do

The first step is declaring your log groups explicitly. By creating them in Terraform before your services run, you eliminate the race condition where a Lambda function tries to write logs and implicitly creates a log group with no policy attached.

Here's a basic pattern:

resource "aws_cloudwatch_log_group" "api_logs" {
  name              = "/aws/lambda/my-api"
  retention_in_days = 30

  tags = {
    Environment = "production"
    Service     = "api"
  }
}

resource "aws_cloudwatch_log_group" "database_logs" {
  name              = "/aws/rds/postgres-prod"
  retention_in_days = 7

  tags = {
    Environment = "production"
    Service     = "database"
  }
}

Notice the naming convention: /aws/lambda/my-api and /aws/rds/postgres-prod. These follow AWS service conventions, making them predictable and easy to discover in the console.

In a real application, you'll want to avoid hardcoding names. Use variables and locals:

locals {
  service_name = "checkout-service"
  environment  = "production"
  log_group_prefix = "/aws/lambda"
}

resource "aws_cloudwatch_log_group" "service_logs" {
  name              = "${local.log_group_prefix}/${local.service_name}-${local.environment}"
  retention_in_days = var.log_retention_days

  tags = {
    Environment = local.environment
    Service     = local.service_name
    ManagedBy   = "Terraform"
  }
}

Now when you link this log group to a Lambda function, IAM role, or ECS task definition, the reference is consistent:

resource "aws_lambda_function" "checkout" {
  filename      = "checkout.zip"
  function_name = "checkout-service-production"
  role          = aws_iam_role.lambda_role.arn
  handler       = "index.handler"

  environment {
    variables = {
      LOG_GROUP = aws_cloudwatch_log_group.service_logs.name
    }
  }

  depends_on = [aws_cloudwatch_log_group.service_logs]
}

The depends_on ensures Terraform creates the log group before the Lambda function, avoiding initialization errors.

Set Retention Days

Retention is where Terraform starts to pay for itself. Without explicit retention, CloudWatch charges you indefinitely for storing logs. With it, you cap costs while maintaining compliance.

The key parameter is retention_in_days. Common values:

In production systems handling payments or user data, you'll often run multiple log groups at different retention tiers. Critical audit logs run 90 days; application logs run 30 days; debug logs run 7 days:

variable "log_retention_tiers" {
  type = map(number)
  default = {
    audit       = 90
    application = 30
    debug       = 7
  }
}

resource "aws_cloudwatch_log_group" "audit_logs" {
  name              = "/aws/lambda/audit-trail"
  retention_in_days = var.log_retention_tiers["audit"]
}

resource "aws_cloudwatch_log_group" "app_logs" {
  name              = "/aws/lambda/app"
  retention_in_days = var.log_retention_tiers["application"]
}

resource "aws_cloudwatch_log_group" "debug_logs" {
  name              = "/aws/lambda/debug"
  retention_in_days = var.log_retention_tiers["debug"]
}

For development and staging, you can set shorter retention or even disable it entirely by omitting the parameter (though this is risky). Using for_each makes this scalable:

resource "aws_cloudwatch_log_group" "environment_logs" {
  for_each = {
    dev  = 1
    stg  = 7
    prod = 30
  }

  name              = "/aws/lambda/app-${each.key}"
  retention_in_days = each.value
}

Encrypt with KMS When Required

By default, CloudWatch encrypts logs at rest using AWS-managed keys. For regulated workloads—payment processing, PII handling, healthcare data—you need customer-managed KMS encryption.

First, create a KMS key:

resource "aws_kms_key" "cloudwatch_logs" {
  description             = "KMS key for CloudWatch logs encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true

  tags = {
    Name = "cloudwatch-logs-key"
  }
}

resource "aws_kms_alias" "cloudwatch_logs" {
  name          = "alias/cloudwatch-logs"
  target_key_id = aws_kms_key.cloudwatch_logs.key_id
}

Next, allow CloudWatch to use the key. CloudWatch has a service principal that needs decrypt/encrypt permissions:

resource "aws_kms_key_policy" "cloudwatch_logs" {
  key_id = aws_kms_key.cloudwatch_logs.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "Enable IAM User Permissions"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        }
        Action   = "kms:*"
        Resource = "*"
      },
      {
        Sid    = "Allow CloudWatch Logs"
        Effect = "Allow"
        Principal = {
          Service = "logs.amazonaws.com"
        }
        Action = [
          "kms:Encrypt",
          "kms:Decrypt",
          "kms:ReEncrypt*",
          "kms:GenerateDataKey*",
          "kms:CreateGrant",
          "kms:DescribeKey"
        ]
        Resource = "*"
      }
    ]
  })
}

Now link the KMS key to your log group:

resource "aws_cloudwatch_log_group" "encrypted_logs" {
  name              = "/aws/lambda/payment-processing"
  retention_in_days = 90
  kms_key_id        = aws_kms_key.cloudwatch_logs.arn

  depends_on = [aws_kms_key_policy.cloudwatch_logs]

  tags = {
    Compliance = "pci-dss"
  }
}

The depends_on is critical here: the key policy must exist before the log group attempts encryption. Without it, Terraform may race and fail.

One thing to watch: KMS encryption increases log write latency by a few milliseconds and adds cost (roughly $0.03 per 10,000 requests). Use it only where compliance requires it, not everywhere.

Subscription Filters to Ship Logs

CloudWatch is a collector, not a warehouse. For long-term analysis, cost optimization, and centralized observability, you need to ship logs to S3, Datadog, Splunk, or an aggregation service. Terraform lets you declare this routing declaratively.

A subscription filter watches a log group and forwards matching logs to a destination. Here's sending logs to Kinesis Firehose, which batches them into S3:

resource "aws_cloudwatch_log_subscription_filter" "to_s3" {
  name            = "logs-to-firehose"
  log_group_name  = aws_cloudwatch_log_group.service_logs.name
  filter_pattern  = "[...]"  # Match all logs
  destination_arn = aws_kinesis_firehose_delivery_stream.logs_to_s3.arn
}

The filter_pattern is powerful. Empty "" or [...] matches everything. You can filter by severity or content:

• "ERROR" – match only lines containing ERROR
• "[timestamp, request_id, status_code >= 400]" – match structured JSON where status >= 400
• "[... , response_time > 1000]" – match slow responses

For a production e-commerce checkout service, you might route errors to Firehose and success logs nowhere:

resource "aws_cloudwatch_log_subscription_filter" "errors_only" {
  name            = "checkout-errors-to-s3"
  log_group_name  = aws_cloudwatch_log_group.checkout_logs.name
  filter_pattern  = "[ERROR]"
  destination_arn = aws_kinesis_firehose_delivery_stream.error_logs_to_s3.arn

  depends_on = [aws_lambda_permission.allow_cloudwatch]
}

resource "aws_lambda_permission" "allow_cloudwatch" {
  statement_id  = "AllowExecutionFromCloudWatch"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.log_processor.function_name
  principal     = "logs.amazonaws.com"
  source_arn    = "${aws_cloudwatch_log_group.checkout_logs.arn}:*"
}

You can also send logs to a Lambda function for real-time processing. This is useful for alerting on critical errors or enriching logs with metadata:

resource "aws_cloudwatch_log_subscription_filter" "to_lambda" {
  name            = "logs-to-processor"
  log_group_name  = aws_cloudwatch_log_group.service_logs.name
  filter_pattern  = "[ERROR] [CRITICAL]"
  destination_arn = aws_lambda_function.log_processor.arn

  depends_on = [aws_lambda_permission.allow_cloudwatch]
}

For SaaS platforms using Datadog or New Relic, many teams use a Lambda wrapper that transforms CloudWatch logs into the vendor's JSON format and ships them via HTTP. Terraform can orchestrate the entire pipeline: log group → subscription filter → Lambda → Datadog.

Common Mistakes

Forgetting the depends_on between log groups and services. If your Lambda function tries to write logs before the log group exists, AWS auto-creates one with default settings. Always add explicit dependency.

Setting retention too low for production. Seven days feels conservative until you're debugging a production incident three days after it started and your logs have already rolled off. Thirty days is a safer baseline for most services.

Not gating KMS encryption behind a variable. You want to enable KMS in production but disable it in development (faster, cheaper). Use a variable and conditional:

resource "aws_cloudwatch_log_group" "app_logs" {
  name              = "/aws/lambda/app"
  retention_in_days = 30
  kms_key_id        = var.enable_kms_encryption ? aws_kms_key.cloudwatch_logs.arn : null
}

Subscription filters without IAM permissions. A subscription filter to Lambda or Firehose requires the service to have permission to invoke the target. Always include the aws_lambda_permission or equivalent.

Over-filtering on subscription filters. If you filter out too many logs, you lose visibility when you need it. Start permissive and tighten over time.

Key Takeaways

• Declare log groups in Terraform before services create them implicitly. This gives you control over retention, encryption, and tagging from day one.
• Set explicit retention days based on use case: 7 days for debug, 30 days for operational, 90 days for compliance. This directly controls costs.
• Use customer-managed KMS encryption for regulated workloads (payment, PII, healthcare). Ensure the key policy grants CloudWatch permissions and add depends_on to avoid races.
• Implement subscription filters to route logs to S3, Firehose, Lambda, or third-party aggregators. Filter patterns let you route only critical data, reducing storage and egress costs.
• Test log group creation and subscription filters in a development stack first. CloudWatch log bugs are often caught only in production without proper integration testing.