Disaster recovery with Terraform: lessons from production

Outages hurt. When production goes down, the speed at which you can rebuild infrastructure determines whether you're talking to customers in minutes or hours. We've learned that disaster recovery with Terraform: lessons from production starts with a fundamental truth—if your infrastructure isn't codified, you can't reliably rebuild it. Over years running systems at scale, we've seen teams recover in under an hour and teams still scrambling after a day. The difference isn't luck; it's discipline.

Code is the blueprint — Everything needed to rebuild lives in the repo

A disaster recovery strategy built on Terraform rests on one principle: your infrastructure code must be the single source of truth. Every resource—VPCs, security groups, databases, load balancers, IAM roles—must exist in version control. Not documented in a Confluence page. Not half-remembered by an engineer. In code.

We've seen situations where teams had Terraform for compute but left networking, IAM, or database configuration in the console. When disaster strikes, those manual resources become recovery blockers. You're hunting through the AWS console, trying to remember why a security group has certain rules, whether that RDS backup is encrypted, or which SNS topics trigger which Lambda functions.

Start by auditing what exists outside Terraform. Use tools like aws ec2 describe-instances or Terraform's import command to bring orphaned resources into code. Make this non-negotiable: nothing manual in production. Every change goes through Terraform, gets reviewed, and lives in Git.

In practice, this means your repository structure should mirror your deployments:

├── environments/
│   ├── production/
│   │   ├── main.tf
│   │   ├── variables.tf
│   │   └── terraform.tfvars
│   ├── staging/
│   └── dr/
├── modules/
│   ├── networking/
│   ├── compute/
│   ├── database/
│   └── monitoring/
└── scripts/
    └── recover.sh

The dr/ environment is not production but a pre-built standby. When disaster strikes, this environment is either already running (in multi-region setups) or can spin up from code in minutes. Your recovery script pulls the latest Terraform configuration, applies it to the DR account, and points traffic over. That's recovery from code.

Multi-region modules — Parameterize region to deploy a standby

Multi-region isn't just about global scale. For disaster recovery, it's your safety net. The architecture works like this: write your modules to accept region as a parameter, then deploy the same infrastructure to a standby region. If primary fails, you promote standby.

Consider a typical application module:

# modules/app/main.tf
variable "region" {
  type = string
}

variable "environment" {
  type = string
}

variable "replica_enabled" {
  type    = bool
  default = false
}

provider "aws" {
  region = var.region
  alias  = "app"
}

resource "aws_ecs_cluster" "main" {
  provider = aws.app
  name     = "${var.environment}-cluster-${var.region}"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

resource "aws_autoscaling_group" "app" {
  provider            = aws.app
  name                = "${var.environment}-asg-${var.region}"
  vpc_zone_identifier = var.subnet_ids
  min_size            = var.replica_enabled ? 1 : 3
  max_size            = var.replica_enabled ? 2 : 10

  tags = [
    {
      key                 = "Name"
      value               = "${var.environment}-app-${var.region}"
      propagate_at_launch = true
    }
  ]
}

At the root level, deploy this module twice:

# environments/production/main.tf
module "app_primary" {
  source = "../../modules/app"

  region           = "us-east-1"
  environment      = "prod"
  subnet_ids       = data.aws_subnets.primary.ids
  replica_enabled  = false
}

module "app_standby" {
  source = "../../modules/app"

  region           = "us-west-2"
  environment      = "prod"
  subnet_ids       = data.aws_subnets.standby.ids
  replica_enabled  = true  # Keep minimal
}

The standby region runs minimal infrastructure—fewer compute instances, scaled-down databases. This keeps costs reasonable while ensuring the infrastructure can spin up quickly. During normal operations, standby stays idle. During failover, you scale it up and update DNS.

One critical detail: test this regularly. We've found teams with multi-region Terraform that haven't actually deployed standby in months. Terraform code can drift. Dependencies might not resolve. Testing catches these issues before they matter.

Data replication — Cross-region database and S3 replication

Infrastructure code gets you running servers and networks fast. But if your data is lost or stranded in the primary region, you're still down. Data replication is non-negotiable for disaster recovery.

For databases, use native cross-region replication. With RDS, configure read replicas in the standby region:

# modules/database/main.tf
variable "replication_enabled" {
  type    = bool
  default = false
}

resource "aws_db_instance" "primary" {
  identifier              = "prod-primary"
  engine                  = "postgres"
  engine_version          = "15.3"
  instance_class          = "db.r6i.2xlarge"
  allocated_storage       = 500
  backup_retention_period = 30
  skip_final_snapshot     = false

  multi_az = true

  tags = {
    Name = "prod-primary-db"
  }
}

resource "aws_db_instance" "replica" {
  count = var.replication_enabled ? 1 : 0

  identifier          = "prod-replica"
  replicate_source_db = aws_db_instance.primary.identifier
  instance_class      = "db.r6i.xlarge"  # Smaller for cost
  skip_final_snapshot = false

  # Replica must be in different region
  provider = aws.replica

  tags = {
    Name = "prod-replica-db"
  }
}

For S3, enable cross-region replication on all buckets holding critical data:

resource "aws_s3_bucket" "primary" {
  bucket = "prod-data-primary"
}

resource "aws_s3_bucket_replication_configuration" "main" {
  depends_on = [aws_s3_bucket_versioning.primary]

  bucket = aws_s3_bucket.primary.id

  role = aws_iam_role.s3_replication.arn

  rule {
    id       = "replicate-to-standby"
    status   = "Enabled"
    priority = 1

    filter {
      prefix = ""
    }

    destination {
      bucket       = aws_s3_bucket.standby.arn
      storage_class = "STANDARD_IA"

      replication_time {
        status = "Enabled"
        time {
          minutes = 15
        }
      }
    }
  }
}

The key metrics: Recovery Time Objective (RTO) is how fast you need to be running again. Recovery Point Objective (RPO) is how much data loss you can tolerate. Cross-region database replicas give you RTO in minutes and RPO of seconds. Cross-region S3 replication gives you similar guarantees for object storage. Make sure both are in code so they survive infrastructure rebuilds.

Practice — Run recovery drills in a test account

Theory and production diverge. Terraform code that looks correct might fail when actually applied. Permissions might be wrong. State files might be in unexpected states. Dependencies might not resolve as expected. The only way to know is to actually run a recovery.

Schedule monthly disaster recovery drills. Pick a test AWS account separate from production. Pull the latest Terraform code. Assume the primary region is gone. Time how long it takes to:

1. Apply Terraform to the DR account to spin up infrastructure
2. Failover databases to standby region
3. Update DNS to point to the new region
4. Validate that traffic flows and services are healthy

This is not a dry run. Actually do it. Log the time. Document what broke and what surprised you.

Here's a sample recovery checklist in code:

# scripts/recover.sh
#!/bin/bash
set -e

TIMESTAMP=$(date +%s)
DR_ACCOUNT="123456789012"
DR_REGION="us-west-2"

echo "Starting DR recovery at $(date)"

# Step 1: Assume role in DR account
aws sts assume-role \
  --role-arn "arn:aws:iam::${DR_ACCOUNT}:role/TerraformRole" \
  --role-session-name "dr-recovery-${TIMESTAMP}"

# Step 2: Apply Terraform to DR environment
cd environments/dr
terraform init
terraform apply -auto-approve -var-file=terraform.tfvars

# Step 3: Wait for databases to be ready
echo "Waiting for database replica to catch up..."
sleep 60

# Step 4: Update Route53 to point to DR
terraform apply -auto-approve -target=aws_route53_record.main

echo "Recovery complete at $(date)"

During the drill, capture lessons. Common findings: state files weren't accessible, IAM permissions were incomplete, dependencies on external services weren't documented, or the standby infrastructure was too small. Fix these in code, not with manual notes.

Common mistakes

Storing credentials in code: Your Terraform code lives in Git, which means it can be compromised. Never commit AWS keys, database passwords, or API tokens. Use AWS Secrets Manager, parameter store, or environment variables instead.

Forgetting to test failover: Infrastructure that hasn't been tested is infrastructure that won't work. Teams often maintain Terraform for standby but never actually spin it up. When disaster strikes, surprises emerge.

**Mismatched versions:** Primary and standby diverge over time. Terraform code for primary gets updated, but DR environment uses an older version. When you need to failover, you're deploying different infrastructure. Use Git branches and CI/CD to keep them in sync.

Ignoring network routing: Your infrastructure can be perfect, but if traffic still routes to the failed region, you're down. Ensure Route53 health checks, load balancer failover, and DNS updates are all codified and tested.

Key takeaways

• Keep all infrastructure in Terraform code, committed to version control. Manual resources become recovery blockers.
• Write modules parameterized by region so the same code deploys to primary and standby environments.
• Implement cross-region replication for databases and S3 buckets to minimize data loss during failover.
• Run monthly disaster recovery drills in a test account. Find problems before production depends on them.
• Document RTO and RPO targets and measure them during drills. Recovery that takes longer than your SLA is not recovery.
• Automate failover steps in scripts that live in the repository. Manual runbooks diverge from reality and fail under pressure.