Backups with Terraform: a practical guide for developers

Introduction

We've all heard the horror stories: a misconfigured security group wipes out production data, a developer runs the wrong SQL migration, or infrastructure gets corrupted by malware. The difference between a Friday afternoon incident and a Monday morning story depends entirely on whether you have a recovery plan. Backups with Terraform: a practical guide for developers means treating your backup strategy as infrastructure code—versioned, tested, and repeatable—rather than something you hope works when disaster strikes.

In this guide, we'll walk through how to build production-grade backup strategies using Terraform, AWS Backup, and cross-region replication. By the end, you'll have a framework to protect databases, file systems, and application state without manual intervention.

AWS Backup Plans and Vaults

AWS Backup is a managed service that centralizes backup policy across multiple AWS services: RDS, DynamoDB, EBS volumes, EFS, and more. Rather than writing custom scripts for each resource type, you define a backup plan once and let AWS handle the scheduling and retention.

A backup vault is where snapshots and backups are stored. You can encrypt vaults with KMS keys, set resource-based policies, and even lock them to prevent accidental deletion. In Terraform, this looks straightforward:

resource "aws_backup_vault" "primary" {
  name        = "prod-backup-vault"
  kms_key_arn = aws_kms_key.backup.arn

  tags = {
    Environment = "production"
    ManagedBy   = "terraform"
  }
}

resource "aws_backup_plan" "daily" {
  name = "prod-daily-backup"

  rule {
    rule_name         = "daily_backups"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 2 * * ? *)"  # 2 AM UTC daily

    lifecycle {
      delete_after = 30
    }

    recovery_point_tags = {
      BackupType = "daily"
      Automated  = "true"
    }
  }

  tags = {
    Environment = "production"
  }
}

resource "aws_backup_selection" "rds_databases" {
  name           = "rds-production-selection"
  plan_id        = aws_backup_plan.daily.id
  iam_role_arn   = aws_iam_role.backup_role.arn
  selection_tag_type = "TAGKEY_AND_VALUE_LENGTH"

  selection_tag {
    type   = "STRINGEQUALS"
    key    = "Backup"
    value  = "daily"
  }

  resources = ["*"]
}

resource "aws_iam_role" "backup_role" {
  name = "aws-backup-service-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "backup.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "backup_policy" {
  role       = aws_iam_role.backup_role.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSBackupServiceRolePolicyForBackup"
}

This setup creates a backup vault, a daily backup plan that retains snapshots for 30 days, and a selection rule that targets any resource tagged with Backup=daily. The IAM role grants AWS Backup service permissions to create and manage snapshots.

Key considerations: use meaningful retention windows based on your RPO (Recovery Point Objective). For production databases, 30 days is a baseline. For compliance-heavy workloads (healthcare, finance), extend to 90 days or longer. Also, never hardcode schedules; parameterize them via variables so different environments can have different cadences.

Tag-Based Resource Selection

Tag-based selection is powerful because it decouples your backup strategy from resource creation. When a new RDS instance or EBS volume is created with the correct tags, it's automatically included in the backup plan. This scales much better than maintaining explicit resource lists.

Here's a practical scenario: you're running multiple microservices, each with its own database. Some services are critical (user auth, payment processing) and need hourly backups. Others are less critical and can tolerate daily backups. Instead of updating the backup plan each time a new service launches, you tag resources and let selection rules handle the logic:

resource "aws_backup_plan" "comprehensive" {
  name = "multi-tier-backup-plan"

  rule {
    rule_name         = "critical_hourly"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 * * * ? *)"  # Every hour
    
    lifecycle {
      delete_after = 7
    }
  }

  rule {
    rule_name         = "standard_daily"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 3 * * ? *)"  # 3 AM UTC daily
    
    lifecycle {
      delete_after = 30
    }
  }
}

resource "aws_backup_selection" "critical_services" {
  name           = "critical-services-hourly"
  plan_id        = aws_backup_plan.comprehensive.id
  iam_role_arn   = aws_iam_role.backup_role.arn

  selection_tag {
    type   = "STRINGEQUALS"
    key    = "BackupTier"
    value  = "critical"
  }

  # Apply the hourly rule only
  selection_rule {
    type   = "StringEquals"
    key    = "aws:ResourceTag/BackupTier"
    value  = "critical"
  }

  resources = ["*"]
}

resource "aws_backup_selection" "standard_services" {
  name           = "standard-services-daily"
  plan_id        = aws_backup_plan.comprehensive.id
  iam_role_arn   = aws_iam_role.backup_role.arn

  selection_tag {
    type   = "STRINGEQUALS"
    key    = "BackupTier"
    value  = "standard"
  }

  resources = ["*"]
}

When your team provisions a new RDS database, they simply add BackupTier = "critical" to its tags. The backup selection rule automatically picks it up. No code changes needed.

This also makes auditing easier. You can query your AWS environment for all resources with BackupTier tags to verify coverage, and use AWS Config rules to detect any databases or volumes that lack proper backup tags.

Cross-Region Copies for Critical Data

A backup stored in the same region as your infrastructure offers protection against accidental deletion or corruption, but not against regional outages (rare, but they happen). For critical systems—databases that power customer-facing features, transaction logs, customer PII—you need cross-region copies.

AWS Backup supports copying recovery points to another region. This is where your infrastructure-as-code approach really shines: you define the copy rule once, and Terraform ensures it's enforced consistently:

resource "aws_backup_vault" "primary" {
  name        = "prod-backup-vault-us-east-1"
  kms_key_arn = aws_kms_key.backup_primary.arn

  tags = {
    Region      = "us-east-1"
    Environment = "production"
  }
}

resource "aws_backup_vault" "replica" {
  provider    = aws.secondary_region  # us-west-2
  name        = "prod-backup-vault-us-west-2"
  kms_key_arn = aws_kms_key.backup_replica.arn

  tags = {
    Region      = "us-west-2"
    Environment = "production"
  }
}

resource "aws_backup_plan" "critical_with_copy" {
  name = "critical-data-backup-with-replication"

  rule {
    rule_name         = "daily_with_cross_region_copy"
    target_vault_name = aws_backup_vault.primary.name
    schedule          = "cron(0 2 * * ? *)"

    lifecycle {
      delete_after = 90
    }

    copy_action {
      destination_vault_arn = aws_backup_vault.replica.arn
      
      lifecycle {
        cold_storage_after = 30
        delete_after       = 365
      }
    }

    recovery_point_tags = {
      CriticalData = "true"
      Region       = "replicated"
    }
  }

  tags = {
    Environment = "production"
  }
}

resource "aws_backup_selection" "critical_rds" {
  name           = "critical-databases"
  plan_id        = aws_backup_plan.critical_with_copy.id
  iam_role_arn   = aws_iam_role.backup_role.arn

  selection_tag {
    type   = "STRINGEQUALS"
    key    = "BackupTier"
    value  = "critical"
  }

  resources = ["*"]
}

Here's what happens: every day at 2 AM UTC, the backup plan creates a recovery point in us-east-1. AWS Backup automatically copies that recovery point to us-west-2. After 30 days, copies move to cold storage (cheaper). After a year, both the original and copy are deleted.

The cost implication is real—cross-region copies roughly double your backup storage cost—so reserve them for truly critical data. A good heuristic: if the data loss would cause customer-facing outages, revenue loss, or compliance violations, it deserves cross-region replication.

Test Restores Regularly

A backup that hasn't been tested is just an expensive storage bill. We've all heard the cautionary tale of teams with years of backups who discovered—when they actually needed to restore—that the backups were corrupted or incomplete. Testing your recovery process is as important as creating the backups.

There are two ways to approach this with Terraform: scheduled restore tests in non-production environments, or manual restore validations. Here's a practical setup:

resource "aws_db_instance" "test_restore_target" {
  identifier          = "prod-db-restore-test"
  instance_class      = "db.t3.micro"
  allocated_storage   = 20
  engine              = "postgres"
  skip_final_snapshot = true

  tags = {
    Name        = "RDS Restore Test"
    Purpose     = "backup-validation"
    Environment = "testing"
  }
}

# Lambda function to periodically restore from backup and validate
resource "aws_lambda_function" "backup_test" {
  filename      = "backup_test_lambda.zip"
  function_name = "backup-restore-validator"
  role          = aws_iam_role.lambda_backup_test.arn
  handler       = "index.handler"
  timeout       = 300

  environment {
    variables = {
      BACKUP_VAULT_NAME     = aws_backup_vault.primary.name
      TEST_RDS_IDENTIFIER   = aws_db_instance.test_restore_target.id
      SNS_TOPIC_ARN         = aws_sns_topic.backup_alerts.arn
    }
  }

  depends_on = [aws_iam_role_policy.lambda_backup_policy]
}

resource "aws_cloudwatch_event_rule" "backup_test_schedule" {
  name                = "monthly-backup-restore-test"
  schedule_expression = "cron(0 4 1 * ? *)"  # 1st of month, 4 AM UTC
}

resource "aws_cloudwatch_event_target" "backup_test_lambda" {
  rule      = aws_cloudwatch_event_rule.backup_test_schedule.name
  target_id = "BackupRestoreTestLambda"
  arn       = aws_lambda_function.backup_test.arn
}

resource "aws_lambda_permission" "allow_eventbridge" {
  statement_id  = "AllowExecutionFromEventBridge"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.backup_test.function_name
  principal     = "events.amazonaws.com"
  source_arn    = aws_cloudwatch_event_rule.backup_test_schedule.arn
}

resource "aws_iam_role" "lambda_backup_test" {
  name = "lambda-backup-restore-test-role"

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Service = "lambda.amazonaws.com"
        }
        Action = "sts:AssumeRole"
      }
    ]
  })
}

resource "aws_iam_role_policy" "lambda_backup_policy" {
  name   = "lambda-backup-restore-policy"
  role   = aws_iam_role.lambda_backup_test.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Action = [
          "backup:DescribeRecoveryPoint",
          "backup:DescribeBackupVault",
          "backup:ListRecoveryPointsByBackupVault",
          "rds-db:DescribeDBInstances",
          "rds:RestoreDBInstanceFromDBSnapshot"
        ]
        Resource = "*"
      },
      {
        Effect = "Allow"
        Action = [
          "sns:Publish"
        ]
        Resource = aws_sns_topic.backup_alerts.arn
      }
    ]
  })
}

This Terraform configuration schedules a monthly Lambda function that attempts to restore the latest backup to a test RDS instance. If the restore succeeds, the instance is cleaned up and an SNS notification is sent. If it fails, you get an alert before you actually need that backup.

For critical applications, go further: after the restore, run automated tests against the restored database (schema validation, data integrity checks, application smoke tests). That way you're not just testing that the backup exists—you're testing that you can actually use it to recover your system.

Common Mistakes

• Assuming backups are created immediately. AWS Backup is asynchronous. If you delete a resource seconds after creating the backup plan, that first backup might not have completed yet. Always verify recovery points exist before decommissioning resources.
• Ignoring IAM permissions. The backup service role needs explicit permissions for each AWS service (RDS, EBS, DynamoDB, etc.). If you copy the backup policy but don't attach it properly, your backups will silently fail for hours or days before you notice.
• Setting retention periods too aggressively. A common trap: "we'll keep backups for 7 days to save money." Then a data corruption issue is discovered on day 8. Set retention based on realistic incident detection and investigation timelines, not just cost.
• Cross-region copies without cost awareness.strong> Replicating backups doubles storage and data transfer costs. If you're replicating all resources, you'll get a surprise bill. Use cross-region copies selectively for truly critical data.
• Not tagging resources for backup inclusion. If you forget to tag a new database, it won't be backed up automatically. This is easy to miss if your team isn't familiar with the tagging strategy. Document it, use resource naming conventions, and consider using AWS Config to enforce tags.
• Testing restores only in emergencies. The first time you test a restore should not be during an outage. Catch configuration issues, permission problems, and data inconsistencies before you need the backup to recover production.

Key Takeaways

• Define backup vaults, plans, and selection rules in Terraform. This ensures your backup strategy is versioned, peer-reviewed, and reproducible across environments.
• Use tag-based selection to automatically include new resources in backup plans. This scales better than managing explicit resource lists and keeps your recovery strategy elastic as infrastructure grows.
• Cross-region copies protect against regional failures for critical data. Balance the protection against the cost of dual-region replication and only apply it where RPO and RTO requirements justify it.
• Schedule regular restore tests—monthly at minimum—using automation (Lambda, EventBridge). Testing in production is expensive; testing in non-production is cheap insurance.
• Monitor backup completion and restoration success with CloudWatch alarms and SNS notifications. Silent backup failures are worse than no backups at all.
• Document your backup retention policy and recovery procedures. When a 2 AM incident happens, you need to know exactly which backups exist, how old they are, and how to use them.