Destroying infrastructure safely: lessons from production

Introduction

We've all been there: staring at a terraform destroy command, finger hovering over Enter, wondering if this is really going to work. Destroying infrastructure safely is not something you learn from a tutorial—it's something you earn through production incidents. Over 98 days of infrastructure-as-code practice, we've built up our environments with care. Now we need to tear them down with equal precision. The difference between a clean teardown and an infrastructure disaster often comes down to a few deliberate safeguards and habits.

terraform plan -destroy first

The single most important habit to build is running terraform plan -destroy before ever executing terraform destroy. This command shows you exactly what will be removed without making any changes. It's your safety net.

Here's what we typically see in production teams:

• Junior engineers skip the plan step and lose a database.
• Experienced teams always review the plan, sometimes twice.

When you run plan -destroy on a real-world stack, you're looking at a detailed list of resources Terraform intends to remove. That list might include:

# Example output from terraform plan -destroy
# aws_db_instance.primary will be destroyed
# aws_s3_bucket.logs will be destroyed
# aws_security_group.app will be destroyed
# aws_vpc.main will be destroyed

Before you approve that plan, ask yourself: do I really want to delete that RDS instance? Is that S3 bucket with two years of log data something we can afford to lose? If the answer is no to any question, your infrastructure code needs adjustment—not your destroy command.

A practical workflow we use across teams:

1. Run terraform plan -destroy -out=destroy.tfplan in your CI/CD environment (not locally).
2. Have another engineer review the plan output in a pull request or chat channel.
3. Only then run terraform apply destroy.tfplan.

This three-step process has prevented more accidents than any other single practice. It turns destruction from a one-person decision into a team decision.

Prevent_destroy on critical resources

Some resources should never be destroyed by Terraform, even if your code says so. That's what the lifecycle.prevent_destroy meta-argument does. It acts as a circuit breaker—Terraform will refuse to destroy that resource, forcing you to manually remove the protection before deletion can proceed.

Use this on anything that would hurt if lost:

resource "aws_db_instance" "primary" {
  identifier     = "production-postgres"
  engine         = "postgres"
  engine_version = "15.3"
  instance_class = "db.m6i.xlarge"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket" "backups" {
  bucket = "company-backups-prod"

  lifecycle {
    prevent_destroy = true
  }
}

When someone tries to destroy a resource with prevent_destroy = true, Terraform stops cold:

Error: Instance cannot be destroyed

  on main.tf line 42, in resource "aws_db_instance" "primary":
   42: resource "aws_db_instance" "primary" {

Resource instance aws_db_instance.primary has lifecycle.prevent_destroy set, 
but the plan calls for this instance to be destroyed. To suppress this error 
and proceed with the destruction anyway, add the -allow-destroy flag when 
re-planning and re-applying.

This forces intentionality. You can't accidentally delete your production database during a late-night refactor. Someone has to explicitly remove the protect, run plan, review it, and apply it. That friction is intentional.

In practice, we tier this protection:

• Production databases and backups: always prevent_destroy
• Production load balancers and DNS: always prevent_destroy
• Development databases: prevent_destroy only if they contain seed data you can't regenerate
• Temporary resources: no protection

Empty buckets and snapshots policy

S3 buckets with objects in them cannot be destroyed by Terraform by default. The same applies to other stateful resources like EBS snapshots, RDS snapshots, and AMIs. Terraform will error out and leave the resource behind. This is actually a safety feature, but it's annoying when you're intentionally cleaning up.

The right approach is to be explicit about cleaning up data before destruction:

resource "aws_s3_bucket" "logs" {
  bucket = "app-logs-${var.environment}"
}

resource "aws_s3_bucket_versioning" "logs" {
  bucket = aws_s3_bucket.logs.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Empty the bucket before destroying it
resource "null_resource" "empty_bucket" {
  provisioner "local-exec" {
    command = "aws s3 rm s3://${aws_s3_bucket.logs.id} --recursive"
    when    = destroy
  }
}

resource "aws_s3_bucket_public_access_block" "logs" {
  bucket = aws_s3_bucket.logs.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

We don't recommend doing this in production, but in non-production environments, it automates the tedious cleanup. More importantly, it documents your intent in code: "when this stack is destroyed, empty the bucket first."

For snapshots and AMIs in development environments, use a separate cleanup job:

• Run a nightly Lambda that deletes snapshots older than 30 days in dev and staging.
• Use AWS Data Lifecycle Manager for automated snapshot retention policies.
• Document the retention policy in your infrastructure repository so the whole team understands it.

Destroy dev environments automatically after hours

One of the safest ways to avoid production accidents is to never keep non-production infrastructure around longer than necessary. Many teams now treat development and staging environments as ephemeral—spin them up for a sprint or feature branch, then destroy them when done.

A simple but powerful pattern is scheduled destruction:

variable "auto_destroy_after_hours" {
  description = "Automatically destroy this stack at this hour (24-hour format). -1 to disable."
  type        = number
  default     = -1
}

resource "aws_cloudwatch_event_rule" "destroy_schedule" {
  count               = var.auto_destroy_after_hours >= 0 ? 1 : 0
  name                = "destroy-stack-${var.environment}"
  description         = "Trigger stack destruction for ${var.environment}"
  schedule_expression = "cron(0 ${var.auto_destroy_after_hours} ? * MON-FRI *)"
}

resource "aws_cloudwatch_event_target" "destroy_lambda" {
  count     = var.auto_destroy_after_hours >= 0 ? 1 : 0
  rule      = aws_cloudwatch_event_rule.destroy_schedule[0].name
  arn       = aws_lambda_function.destroyer.arn
  role_arn  = aws_iam_role.eventbridge.arn
}

With this pattern, your development stacks spin down every evening at, say, 6 PM on weekdays. Engineers come in the next morning, run terraform apply to bring the stack back up, and everything is fresh. This achieves two things:

1. It cuts cloud costs dramatically (dev environments aren't running idle at 3 AM).
2. It regularly exercises your Terraform code's ability to recreate infrastructure from scratch—finding bugs before production needs it.

Make sure the scheduled destruction uses the same terraform plan -destroy workflow, not a shortcut. Log all destruction events to CloudWatch or your logging system so you can audit what was deleted and when.

Common mistakes

• Running destroy without plan first: Always preview before you delete. There's no "undo" button on cloud infrastructure.
• Forgetting about dependencies: If you destroy a VPC, the security groups and network interfaces inside it get destroyed too. Plan captures these, so review carefully.
• Not using prevent_destroy on anything important: Treating all resources equally in terms of deletion risk is a losing strategy. Protect your most critical assets.
• Destroying production during a deploy: Use separate AWS accounts or workspaces for environments. A Terraform workspace pointing to dev should never accidentally point to prod.
• Ignoring orphaned resources: After a failed destroy, some resources remain outside of Terraform's state. These cost money and create technical debt. Always clean them up.

Key takeaways

• Always run terraform plan -destroy and review the output before running destroy. Have another person review it if possible.
• Use lifecycle.prevent_destroy on production databases, backups, and any stateful resource you can't afford to lose.
• Automate bucket emptying and data cleanup in code for non-production environments, documenting your destruction intent.
• Make development and staging environments ephemeral by scheduling automatic destruction after hours, reducing costs and surfacing Terraform bugs early.
• Treat destruction as a first-class operation in your infrastructure code, not an afterthought. What gets created must also be destroyed predictably.