S3 buckets with Terraform for Terraform & IaC developers

Introduction

S3 is everywhere in AWS workloads, yet we often treat bucket creation as a one-time setup task and move on. When you're writing S3 buckets with Terraform for Terraform & IaC developers, the real challenge isn't spinning up storage—it's baking security, compliance, and operational efficiency into your infrastructure code from day one. In this article, we'll walk through the patterns that teams in production use to prevent data leaks, enforce encryption, maintain audit trails, and manage storage costs automatically.

Why S3 bucket configuration matters

A misconfigured S3 bucket has become the default path to a security incident. Public access, unencrypted data, unlimited retention—these aren't exotic edge cases, they're the norm when developers treat buckets as temporary scratch space. The moment that bucket becomes production-critical, or holds customer data, or stores audit logs, the stakes change. Terraform lets us codify the right defaults so we don't have to remember them every time.

We're not just talking about compliance checkboxes. Proper S3 configuration:

• Prevents accidental public exposure of sensitive data
• Ensures data is encrypted at rest by default
• Creates an immutable audit trail through versioning
• Automatically removes old objects to control costs

Block public access

Public access to S3 buckets remains one of the top sources of data breaches. AWS provides Block Public Access (BPA) settings that act as a safety net—they override IAM policies and bucket policies that might accidentally expose your data. When you enable BPA, even if someone misconfigures a bucket policy or accidentally sets an object ACL to public, the block will still prevent external access.

Here's the pattern we use in production:

resource "aws_s3_bucket" "app_data" {
  bucket = "my-app-data-${data.aws_caller_identity.current.account_id}"
}

resource "aws_s3_bucket_public_access_block" "app_data" {
  bucket = aws_s3_bucket.app_data.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

data "aws_caller_identity" "current" {}

This configuration blocks all four vectors of public access. The account ID suffix prevents bucket name collisions across environments and keeps resource names descriptive. We've found that enabling BPA on every bucket, as a baseline, eliminates a whole category of incidents. Teams that try to be selective about which buckets get BPA inevitably forget on a new bucket and regret it later.

The block_public_acls and ignore_public_acls flags work together: the first prevents new public ACLs, the second ignores existing ones. The policy variants do the same for bucket policies. This redundancy is intentional—it's defense in depth.

Server-side encryption

Encryption at rest in S3 is not optional in modern deployments. AWS offers two main approaches: SSE-S3 (managed by AWS) and SSE-KMS (managed by your own AWS KMS key). SSE-S3 is simpler and free. SSE-KMS gives you control over key rotation and audit logs, at the cost of a small per-request charge and operational overhead. For most teams, SSE-KMS on sensitive buckets and SSE-S3 elsewhere is the right balance.

Here's how to enforce encryption with Terraform:

resource "aws_s3_bucket_server_side_encryption_configuration" "app_data" {
  bucket = aws_s3_bucket.app_data.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
    bucket_key_enabled = true
  }
}

# For sensitive data, use KMS instead:
resource "aws_kms_key" "s3" {
  description             = "KMS key for S3 encryption"
  deletion_window_in_days = 10
  enable_key_rotation     = true
}

resource "aws_kms_alias" "s3" {
  name          = "alias/s3-encryption"
  target_key_id = aws_kms_key.s3.key_id
}

resource "aws_s3_bucket_server_side_encryption_configuration" "sensitive_data" {
  bucket = aws_s3_bucket.sensitive.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm      = "aws:kms"
      kms_master_key_id  = aws_kms_key.s3.arn
    }
    bucket_key_enabled = true
  }
}

Setting bucket_key_enabled = true reduces KMS API calls and cost. Without it, every PUT and GET operation makes a separate KMS request; with it, S3 derives a temporary data key from the KMS master key, amortizing the cost. In a high-volume bucket, this setting alone can reduce your KMS bill significantly.

The key detail: encryption with SSE-S3 is invisible to the application. Objects are encrypted and decrypted transparently. With SSE-KMS, you must ensure your application has permission to use the KMS key, otherwise PUTs and GETs will fail silently with permission errors in CloudTrail.

Versioning for important data

Versioning turns your S3 bucket into a time machine. Every time you overwrite or delete an object, S3 keeps the previous version. This is critical for audit logs, configuration backups, or any data you might need to recover. The cost is storage: you pay for every version, not just the latest one. That's why versioning pairs naturally with lifecycle rules (which we'll cover next).

Enable versioning with a single resource:

resource "aws_s3_bucket_versioning" "app_data" {
  bucket = aws_s3_bucket.app_data.id

  versioning_configuration {
    status     = "Enabled"
    mfa_delete = false  # Set to true only if you really need it
  }
}

# Alternative for less-critical buckets: Suspended
# (keeps current version, but doesn't create new versions on overwrites)
resource "aws_s3_bucket_versioning" "temp_data" {
  bucket = aws_s3_bucket.temp.id

  versioning_configuration {
    status = "Suspended"
  }
}

Once versioning is enabled, you cannot disable it—only suspend it. Suspension stops creating new versions but keeps existing ones. This is a one-way door, so think about it before enabling on buckets with millions of small objects.

A common real-world scenario: a bug in your data pipeline writes corrupt data to a bucket for six hours before an alert fires. With versioning and a recent snapshot, you can restore the bucket to a known-good state. Without it, you're stuck. We've seen teams recover from this situation; we've also seen teams rebuild datasets from scratch because they didn't have versioning.

Lifecycle rules for old objects

Lifecycle rules automate the deletion or transition of objects based on age or tags. This is where you control storage costs without manual intervention. The pattern is: logs and temporary files go to S3 Standard initially, transition to Glacier after 30 days, delete after 90 days. Critical backups stay in Standard forever, or move to cheaper tiers on a longer schedule.

Here's a realistic lifecycle configuration:

resource "aws_s3_bucket_lifecycle_configuration" "app_logs" {
  bucket = aws_s3_bucket.logs.id

  rule {
    id     = "expire-old-logs"
    status = "Enabled"

    filter {
      prefix = "application-logs/"
    }

    transition {
      days          = 30
      storage_class = "STANDARD_IA"  # Infrequent access
    }

    transition {
      days          = 90
      storage_class = "GLACIER"
    }

    expiration {
      days = 365
    }
  }

  rule {
    id     = "delete-incomplete-multipart"
    status = "Enabled"

    filter {}

    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "backups" {
  bucket = aws_s3_bucket.backups.id

  rule {
    id     = "archive-old-backups"
    status = "Enabled"

    filter {
      prefix = "daily-backups/"
    }

    transition {
      days          = 7
      storage_class = "DEEP_ARCHIVE"
    }

    noncurrent_version_expiration {
      noncurrent_days = 30
    }
  }
}

The second rule—cleaning up incomplete multipart uploads—deserves attention. If an application crashes mid-upload, S3 leaves orphaned fragments that still consume storage. By default, they're never cleaned up. Setting a 7-day expiration on these fragments prevents surprise storage bills from failed uploads.

The noncurrent_version_expiration rule works with versioning. It deletes old versions after a certain age, letting you keep recent versions for recovery but not pay for years of history. This is the cost-effective way to have both versioning and reasonable storage bills.

Storage class transitions incur a small cost per object and have minimum storage durations (Standard-IA requires 30 days minimum, Glacier requires 90 days). Don't transition objects that will be deleted in 14 days—you'll pay the transition cost plus the minimum duration fee, losing money. Plan your lifecycle windows accordingly.

Common mistakes

Forgetting to apply Public Access Block

Teams often create the bucket and encryption resources, then skip BPA because "we have a good IAM policy." This overconfidence is expensive. BPA costs nothing and catches human error. Always include it.

Enabling versioning without lifecycle rules

Versioning is free; storing unlimited versions is not. We've seen buckets blow past budget because they held years of versions and no one cleaned them up. Use noncurrent_version_expiration to cap history age.

Using SSE-S3 for highly sensitive data

SSE-S3 is fine for most workloads, but audit-sensitive environments (financial records, healthcare data) should use SSE-KMS so key usage is logged in CloudTrail. The distinction matters for compliance.

Lifecycle transitions that violate minimum storage durations

Transitioning a 10-day-old object to Glacier (90-day minimum) means you pay for 90 days even though the object is deleted after 20. Read the storage class requirements and calculate your break-even point.

Mixing concerns in a single bucket

Logs, temporary files, and permanent backups shouldn't live in the same bucket. They have different lifecycle needs and security requirements. Use separate buckets and reference them clearly in your Terraform module names.

Putting it together: a complete example

Here's a module we use for production application data:

resource "aws_s3_bucket" "app_storage" {
  bucket = "app-storage-${data.aws_caller_identity.current.account_id}-${var.environment}"
  
  tags = {
    Environment = var.environment
    Purpose     = "Application data"
  }
}

resource "aws_s3_bucket_public_access_block" "app_storage" {
  bucket                  = aws_s3_bucket.app_storage.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_s3_bucket_server_side_encryption_configuration" "app_storage" {
  bucket = aws_s3_bucket.app_storage.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_versioning" "app_storage" {
  bucket = aws_s3_bucket.app_storage.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_lifecycle_configuration" "app_storage" {
  bucket = aws_s3_bucket.app_storage.id

  rule {
    id     = "archive-old-versions"
    status = "Enabled"

    noncurrent_version_transition {
      noncurrent_days = 30
      storage_class   = "STANDARD_IA"
    }

    noncurrent_version_expiration {
      noncurrent_days = 90
    }
  }
}

This gives us a secure, auditable, cost-controlled bucket suitable for production. The pattern is repeatable and enforceable across teams.

Key takeaways

• Block public access on every bucket—it's free insurance against misconfiguration and catches mistakes that IAM policies miss.
• Encrypt by default—use SSE-S3 for most workloads, SSE-KMS for audit-sensitive data, and always enable bucket key optimization.
• Version for recovery, but expire versions—combine versioning with noncurrent version expiration to get audit trails without infinite storage costs.
• Automate lifecycle transitions—move old data to cheaper storage classes and delete what you don't need, reducing surprise bills.
• Separate concerns—use different buckets for logs, backups, and application data so lifecycle rules don't conflict.
• Test before production—lifecycle rules run once per day, so mistakes take 24 hours to manifest. Validate your rules on a dev bucket first.