SQS and SNS with Terraform: patterns, pitfalls and fixes

SQS and SNS with Terraform: patterns, pitfalls and fixes

Messaging infrastructure feels simple until it silently drops messages or processes them twice. When we deploy SQS and SNS with Terraform—especially across teams and environments—small configuration mistakes cascade into hard-to-debug production issues. This guide walks through the patterns that work, the pitfalls we've hit, and the Terraform fixes that keep our systems reliable.

Why messaging infrastructure matters in IaC

Messaging queues are often the hidden backbone of distributed systems: they decouple services, absorb traffic spikes, and provide retry logic when things fail. But because they're "hidden," teams often treat them as afterthoughts in infrastructure code. The result is inconsistent queue configurations, missing dead-letter queues, visibility timeouts that don't match processing time, and encryption settings that only exist in some environments.

Terraform forces us to be explicit. That's good. It means we can encode years of operational experience into reusable modules, catch drift before it becomes an incident, and ensure every queue—whether in dev or prod—follows the same safety patterns.

Queues with dead-letter queues

The most common pattern we see is a primary SQS queue that receives messages, and a dead-letter queue (DLQ) that captures messages that fail after a maximum number of retries. Without a DLQ, failed messages either loop forever or disappear silently. With one, we have a place to inspect what went wrong.

Here's how we structure it in Terraform:

resource "aws_sqs_queue" "order_dlq" {
  name                      = "${var.environment}-order-dlq"
  message_retention_seconds = 1209600  # 14 days
  
  tags = {
    Environment = var.environment
    Purpose     = "dead-letter-queue"
  }
}

resource "aws_sqs_queue" "order_queue" {
  name                      = "${var.environment}-order-queue"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 345600  # 4 days
  
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_dlq.arn
    maxReceiveCount     = 3
  })
  
  tags = {
    Environment = var.environment
    Purpose     = "primary-queue"
  }
}

output "queue_arn" {
  value = aws_sqs_queue.order_queue.arn
}

output "dlq_arn" {
  value = aws_sqs_queue.order_dlq.arn
}

The redrive_policy is critical. It tells SQS: if a consumer receives a message three times and doesn't delete it (because processing failed), send it to the DLQ. The DLQ's longer retention window (14 days vs. 4 days for the primary queue) gives our team time to investigate and replay messages if needed.

In production, we also monitor the DLQ depth. When messages start piling up there, it's a signal that our consumer is broken and needs attention—often before customers report an issue.

Visibility timeout greater than processing time

Visibility timeout is the window during which a message is invisible to other consumers after one consumer receives it. If a consumer takes longer to process than the timeout allows, another consumer will grab the same message, and you'll process it twice.

We've seen this play out: a visibility timeout of 30 seconds, a consumer that takes 45 seconds to write to a database and call a payment API, and suddenly every order charges twice.

The fix is straightforward but requires honest conversation with your team about how long processing actually takes—not how long you hope it takes.

resource "aws_sqs_queue" "payment_queue" {
  name                       = "${var.environment}-payment-queue"
  visibility_timeout_seconds = 600  # 10 minutes
  message_retention_seconds  = 345600
  
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.payment_dlq.arn
    maxReceiveCount     = 3
  })
  
  tags = {
    Environment = var.environment
    ProcessingTime = "up to 8 minutes"
  }
}

# DLQ for payment failures
resource "aws_sqs_queue" "payment_dlq" {
  name                      = "${var.environment}-payment-dlq"
  message_retention_seconds = 1209600
}

A good rule of thumb: set visibility timeout to 1.5–2x your P99 processing time. If processing might take 8 minutes in peak load, set visibility timeout to 15 minutes. If your consumer crashes halfway through, the message becomes visible again after 15 minutes and another consumer can retry.

We also add tags documenting the expected processing time. When someone updates the visibility timeout months later, they'll understand why it's set that way.

SNS topics fan out to multiple SQS queues

SNS (Simple Notification Service) is the pub-sub layer; SQS is the durable queue. A common pattern is an SNS topic that publishes events (like "order created"), and multiple SQS queues subscribed to that topic—one for the fulfillment service, one for the billing service, one for analytics. Each service gets its own queue, so if one consumer is slow or crashes, others aren't blocked.

Here's how we wire it up:

resource "aws_sns_topic" "order_events" {
  name = "${var.environment}-order-events"
  
  tags = {
    Environment = var.environment
  }
}

resource "aws_sqs_queue" "fulfillment_queue" {
  name                      = "${var.environment}-fulfillment-queue"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 345600
  
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.fulfillment_dlq.arn
    maxReceiveCount     = 3
  })
}

resource "aws_sqs_queue" "fulfillment_dlq" {
  name                      = "${var.environment}-fulfillment-dlq"
  message_retention_seconds = 1209600
}

resource "aws_sns_topic_subscription" "fulfillment" {
  topic_arn = aws_sns_topic.order_events.arn
  protocol  = "sqs"
  endpoint  = aws_sqs_queue.fulfillment_queue.arn
}

# Allow SNS to send messages to the queue
resource "aws_sqs_queue_policy" "fulfillment_policy" {
  queue_url = aws_sqs_queue.fulfillment_queue.id
  
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "sns.amazonaws.com"
        }
        Action   = "sqs:SendMessage"
        Resource = aws_sqs_queue.fulfillment_queue.arn
        Condition = {
          ArnEquals = {
            "aws:SourceArn" = aws_sns_topic.order_events.arn
          }
        }
      }
    ]
  })
}

The queue policy is essential: it allows the SNS topic to actually deliver messages to the queue. Without it, the subscription exists but messages never arrive. We've seen this slip through code review because the policy looks like boilerplate.

For each subscriber (fulfillment, billing, analytics), we repeat the queue + DLQ + policy pattern. Each service has its own queue with its own visibility timeout and retry logic, tuned to that service's needs.

Encryption at rest

Queues can hold sensitive data: payment details, PII, authentication tokens. We always encrypt them at rest using AWS KMS keys. Terraform makes this straightforward but easy to forget in lower environments, leading to dev-prod parity issues.

resource "aws_kms_key" "sqs_key" {
  description             = "KMS key for SQS encryption (${var.environment})"
  deletion_window_in_days = 7
  enable_key_rotation     = true
  
  tags = {
    Environment = var.environment
  }
}

resource "aws_kms_alias" "sqs_key_alias" {
  name          = "alias/${var.environment}-sqs-key"
  target_key_id = aws_kms_key.sqs_key.key_id
}

resource "aws_sqs_queue" "order_queue" {
  name                       = "${var.environment}-order-queue"
  visibility_timeout_seconds = 300
  message_retention_seconds  = 345600
  
  kms_master_key_id = aws_kms_key.sqs_key.id
  
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_dlq.arn
    maxReceiveCount     = 3
  })
  
  tags = {
    Environment = var.environment
  }
}

resource "aws_sqs_queue" "order_dlq" {
  name                      = "${var.environment}-order-dlq"
  message_retention_seconds = 1209600
  kms_master_key_id         = aws_kms_key.sqs_key.id
}

We use the same KMS key for both the primary queue and the DLQ. This keeps it simple and ensures consistent encryption. We also enable key rotation so the key material changes annually without our intervention.

For SNS topics, encryption is configured similarly:

resource "aws_sns_topic" "order_events" {
  name              = "${var.environment}-order-events"
  kms_master_key_id = aws_kms_key.sqs_key.id
  
  tags = {
    Environment = var.environment
  }
}

Common mistakes

• Forgetting the SQS queue policy: The subscription looks correct, but messages never arrive. Always include the queue policy that grants SNS permission to send.
• Visibility timeout equals processing time: Set it higher. A timeout that equals processing time is a race condition waiting to happen. Use 1.5–2x the P99 time.
• No DLQ or maxReceiveCount too high: If a message will never be processed (consumer is broken), you want to know quickly. A low maxReceiveCount (2–3) and an alert on DLQ depth catches issues fast.
• Encryption inconsistent across environments: Dev has no encryption, prod does. This leads to surprises when code written and tested in dev breaks in prod. Use the same Terraform module for all environments and only parameterize what differs (key rotation, retention).
• Missing tags or documentation: Months later, someone asks: "Why is the visibility timeout 900 seconds?" If you haven't tagged it or commented it, that answer is lost.

Key takeaways

• Always pair a primary SQS queue with a dead-letter queue. Set maxReceiveCount low (2–3) so failures surface quickly.
• Set visibility timeout to 1.5–2x your P99 message processing time. This prevents double processing when consumers are slow.
• Use SNS-to-SQS fan-out when multiple services need the same event. Each subscriber gets its own queue with its own retry and timeout settings.
• Encrypt queues and topics at rest with KMS. Use the same key for related queues and topics. Enable key rotation.
• Document expected processing times and retry logic in tags and comments. Future maintainers will thank you.
• Test queue behavior end-to-end in your lowest environment. Visibility timeouts and DLQ logic only reveal problems under load.