Remote state with S3 and DynamoDB locking for production

When your Terraform project moves beyond a single developer's laptop, state management becomes a production concern. Local state files invite merge conflicts, accidental deletions, and the nightmare scenario where two team members run plan and apply simultaneously. Remote state with S3 and DynamoDB locking for production solves this by centralizing state in AWS, enabling safe concurrent access and giving you the audit trail you need for compliance.

Why remote state matters

Local state works fine for experiments, but production infrastructure demands a different approach. Consider a typical scenario: your DevOps team needs to provision a load balancer, your platform team needs to update security groups, and your database administrator needs to scale RDS—all from separate machines, all using the same Terraform root module. Without remote state, you're running blind. One person's changes overwrite another's. State files get lost when laptops die. Nobody knows who changed what, or when.

Remote state backed by S3 with locking through DynamoDB (or S3 native locking) gives you a single source of truth, prevents concurrent modifications, and leaves an audit trail. It's the foundation of any mature Terraform workflow.

Configure the backend — state in S3, locks in DynamoDB

Terraform's backend configuration tells it where and how to store state. The S3 backend is the AWS standard. Here's a production-ready configuration:

terraform {
  backend "s3" {
    bucket           = "acme-tf-state-prod"
    key              = "prod/network.tfstate"
    region           = "ap-south-1"
    encrypt          = true
    dynamodb_table   = "terraform-locks"
  }
}

Let's break down each field:

• bucket: The S3 bucket holding your state files. Use a naming convention that includes your organization and environment.
• key: The path within the bucket. We use prod/network.tfstate to separate environments and components. This same bucket can hold state for dev, staging, and prod.
• region: The AWS region where the bucket lives. Keep this consistent with your primary infrastructure region.
• encrypt: Enables server-side encryption at rest. Always enable this for state containing secrets.
• dynamodb_table: The table name for distributed locking. Terraform will create lock entries here during operations.

Before running terraform init with this backend, you need to create the S3 bucket and DynamoDB table. Here's the bootstrap Terraform for that:

resource "aws_s3_bucket" "terraform_state" {
  bucket = "acme-tf-state-prod"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_locks" {
  name           = "terraform-locks"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

Run this bootstrap stack once in your AWS account, ideally with an admin role. After it completes, initialize your main Terraform configuration with the remote backend. Terraform will detect the backend config and prompt you to migrate existing state if necessary.

Versioning — recover old state when needed

State files change constantly. A misconfigured resource, an accidental deletion, or a botched apply can corrupt your state. Enabling versioning on the S3 bucket lets you roll back to any previous version.

We've already included versioning in the bootstrap code above, but here's what it does:

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  versioning_configuration {
    status = "Enabled"
  }
}

With versioning enabled, every time Terraform writes a new state file, S3 keeps the old one under a version ID. If something goes catastrophically wrong—say an engineer accidentally destroys your production database in the state—you can recover by pulling the previous version.

To restore an old state version from the AWS CLI:

aws s3api get-object \
  --bucket acme-tf-state-prod \
  --key prod/network.tfstate \
  --version-id abc123xyz \
  prod-network-backup.tfstate

Then temporarily point your backend to that file, run terraform refresh to validate it, and decide whether to restore. Never blindly restore state without understanding what changed.

Access control — restrict who can read and write state

State files contain secrets: database passwords, API keys, encryption keys. Access must be strictly controlled. Only your CI/CD pipeline and designated admins should be able to write state. Developers should generally not have direct access.

Implement this with an S3 bucket policy and IAM roles. Here's an example that allows only a CI/CD role and a Terraform admin role:

resource "aws_s3_bucket_policy" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "DenyUnencryptedObjectUploads"
        Effect = "Deny"
        Principal = "*"
        Action = "s3:PutObject"
        Resource = "${aws_s3_bucket.terraform_state.arn}/*"
        Condition = {
          StringNotEquals = {
            "s3:x-amz-server-side-encryption" = "AES256"
          }
        }
      },
      {
        Sid    = "AllowCICDRole"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:role/cicd-terraform-role"
        }
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:DeleteObject"
        ]
        Resource = "${aws_s3_bucket.terraform_state.arn}/*"
      },
      {
        Sid    = "AllowTerraformAdmins"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::123456789012:role/terraform-admin"
        }
        Action = [
          "s3:GetObject",
          "s3:PutObject",
          "s3:ListBucketVersions"
        ]
        Resource = [
          aws_s3_bucket.terraform_state.arn,
          "${aws_s3_bucket.terraform_state.arn}/*"
        ]
      }
    ]
  })
}

And an IAM policy for the CI/CD role to access the DynamoDB lock table:

resource "aws_iam_role_policy" "cicd_terraform" {
  name   = "cicd-terraform-policy"
  role   = aws_iam_role.cicd_terraform.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "dynamodb:DescribeTable",
          "dynamodb:GetItem",
          "dynamodb:PutItem",
          "dynamodb:DeleteItem"
        ]
        Resource = aws_dynamodb_table.terraform_locks.arn
      }
    ]
  })
}

This approach means developers never touch the state directly. They commit code, a CI pipeline validates and applies it. The pipeline runs with the limited CI/CD role, preventing accidental or malicious state corruption.

One state per component — smaller blast radius

A common mistake is storing everything in one giant state file. A root module managing networking, databases, Kubernetes, and application resources becomes bloated. A single typo can affect dozens of unrelated resources. Tests and plans slow down as state grows.

Instead, split your infrastructure into logical components, each with its own state file. For example:

• prod/network.tfstate — VPC, subnets, routing
• prod/database.tfstate — RDS, security groups for database
• prod/eks-cluster.tfstate — EKS control plane
• prod/eks-addons.tfstate — EBS CSI, VPC CNI
• prod/app.tfstate — Application deployments

Each gets its own backend configuration and can be managed independently. If you need to reference output from one component in another, use data sources or Terraform remote state data sources:

data "terraform_remote_state" "network" {
  backend = "s3"

  config = {
    bucket = "acme-tf-state-prod"
    key    = "prod/network.tfstate"
    region = "ap-south-1"
  }
}

resource "aws_eks_cluster" "main" {
  name     = "production"
  role_arn = aws_iam_role.eks_service_role.arn

  vpc_config {
    subnet_ids = data.terraform_remote_state.network.outputs.private_subnet_ids
  }
}

This decoupling means you can update networking without risking the application layer. Plans are faster, diffs are clearer, and rollbacks affect fewer resources.

Common mistakes

Forgetting to initialize the backend before team use: If you add a backend config but don't run terraform init first, teammates will still be using local state. Coordinate backend migrations carefully—have one person initialize, commit the lock file if you use one, and have others run terraform init to migrate.

Mixing environments in one bucket without strict key prefixes: If dev and prod state files sit in the same bucket with similar key names, someone will inevitably run terraform destroy on the wrong one. Use a clear naming scheme: dev/`, `staging/`, `prod/ as top-level prefixes.

Not enabling MFA delete: For extra safety on the state bucket, enable MFA delete on versioning. This requires an MFA token to permanently delete object versions, protecting against accidental or malicious data loss.

Sharing credentials instead of using IAM roles: Never share AWS credentials for state access. Always use IAM roles—in CI/CD systems, on EC2 instances, or via temporary STS tokens. Roles are auditable, rotatable, and follow least-privilege principles.

Manual state edits: Terraform state is not meant for hand editing. If you must rescue corrupted state, use terraform state commands or restore from versioning, never edit the JSON directly.

Key takeaways

• Remote state in S3 with DynamoDB locking is the foundation of team-based Terraform workflows.
• Always encrypt state at rest and in transit, and restrict access via IAM and bucket policies.
• Enable S3 versioning to recover from state corruption or accidental deletions.
• Split infrastructure into logical components with separate state files to reduce blast radius and improve plan speed.
• Use CI/CD pipelines as the only mechanism for writing state, eliminating manual changes and configuration drift.
• Reference remote state outputs between components using terraform_remote_state data sources rather than copying values.