100 days of Terraform and IaC: the recap for production

100 days of Terraform and IaC: the recap for production

After 100 days of working with Terraform and infrastructure as code, the difference between experiments and production-grade deployments becomes unmistakable. We've learned that infrastructure automation isn't just about writing HCL—it's about building systems that survive real-world chaos, scale with your team, and keep costs predictable. This recap covers the lessons that matter most when you're responsible for systems people depend on.

State is the heart — Remote, locked, encrypted and split by component

Terraform state is your source of truth for infrastructure. Store it locally during development, and your first production incident will teach you why that's a mistake. We've seen teams lose hours to corrupted local state files, conflicting edits, and the nightmare of recreating resources mid-deployment.

Remote state must be the default. Use S3 with versioning and server-side encryption, DynamoDB for locking, or managed services like Terraform Cloud. Here's why locking matters: without it, two engineers can run terraform apply simultaneously, and the second one will overwrite the first's changes in state. DynamoDB prevents this. Encryption ensures that sensitive values—database passwords, API keys, RDS master usernames—aren't readable in S3.

Split state by component. A single monolithic state file for your entire infrastructure becomes a bottleneck. We structure ours with separate state files for networking, Kubernetes clusters, databases, and application deployments. This means:

• One team can deploy without blocking another
• Plan and apply operations complete faster
• Blast radius is contained if state corruption happens
• Different teams can own different components

Use remote state references with terraform_remote_state data sources to wire them together. Your application state imports outputs from the networking state, which outputs VPC and subnet IDs:

data "terraform_remote_state" "networking" {
  backend = "s3"
  config = {
    bucket = "my-org-terraform-state"
    key    = "production/networking/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_security_group" "app" {
  vpc_id = data.terraform_remote_state.networking.outputs.vpc_id
  
  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = [data.terraform_remote_state.networking.outputs.app_subnet_cidr]
  }
}

Encrypt sensitive values in state using sensitive = true on outputs and resources. Document where state is stored, who has access, and how to recover from corruption. We keep a runbook for state recovery in our internal wiki.

Modules create leverage — Small, versioned and well-documented

Copy-pasting Terraform code is a trap that catches every team exactly once. After your third RDS cluster, you'll have three slightly different configurations, and updates become nightmares. Modules solve this. A well-designed module is a contract: you define inputs, we handle the implementation details, you get predictable outputs.

Keep modules small and focused. A module that provisions networking, Kubernetes, a database, and monitoring is a module that's impossible to reuse. We structure ours around single responsibilities: rds-postgres, eks-cluster, s3-private-bucket, iam-service-role. Small modules are easier to test, document, and maintain.

Version your modules. Store them in a Git repository with semantic versioning. Reference a specific version in your root modules, not the default branch. This prevents production deployments from breaking when a colleague makes changes:

module "rds_primary" {
  source = "git::https://git.company.internal/terraform-modules/rds-postgres.git?ref=v2.1.4"
  
  instance_class       = "db.r6i.xlarge"
  allocated_storage    = 100
  backup_retention_days = 30
  multi_az             = true
  
  subnet_group_name = aws_db_subnet_group.private.name
  security_group_ids = [aws_security_group.rds.id]
}

Document inputs, outputs, and assumptions. Write a README.md in each module repository. Include example usage, required IAM permissions, and any manual steps after Terraform runs. We've found that a one-minute README prevents a 30-minute Slack conversation later.

Test modules before tagging a version. Use terraform validate, terraform fmt, and a tool like Terratest or Terraform Cloud's cost estimation. Catch configuration mistakes in version control, not in production.

Automate the workflow — Plans in PRs, applies from CI with OIDC

Manual terraform apply from a laptop has no place in production infrastructure. The moment it's time-critical or high-stakes, someone runs it tired, makes a mistake, and pages wake up at 2 AM.

Every change starts with a pull request. When a PR is opened, your CI pipeline runs terraform plan on the affected directories and posts the plan as a comment. Engineers review what will actually change before approval. This catches configuration mistakes, security issues, and unintended resource deletions.

We use GitHub Actions with OpenID Connect (OIDC) for authentication. No static AWS credentials in CI—instead, GitHub issues short-lived OIDC tokens that assume a Terraform CI role in AWS. Configure it like this:

resource "aws_iam_role" "github_actions" {
  name = "github-actions-terraform"
  
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::ACCOUNT_ID:oidc-provider/token.actions.githubusercontent.com"
        }
        Action = "sts:AssumeRoleWithWebIdentity"
        Condition = {
          StringLike = {
            "token.actions.githubusercontent.com:sub" = "repo:YOUR_ORG/*:ref:refs/heads/main"
          }
        }
      }
    ]
  })
}

resource "aws_iam_role_policy" "github_terraform" {
  role   = aws_iam_role.github_actions.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Effect   = "Allow"
        Action   = ["ec2:*", "rds:*", "s3:*"]
        Resource = "*"
      }
    ]
  })
}

On merge, apply runs automatically from CI. No manual step. Log every apply to a central audit trail—who merged, when, what changed, and the Terraform run ID. This becomes invaluable when you're investigating why a service behaves unexpectedly.

Use environments in your repository for separate workflows: plan in a staging environment before production applies. Require an approval step for production. Limit who can approve: usually a senior engineer or on-call lead.

Guardrails scale teams — Policies, scans, tagging and cost checks

Without guardrails, infrastructure sprawls. Developers create resources without tags, databases are set to publicly accessible by mistake, and EC2 instances run for months forgotten. Guardrails prevent mistakes at the policy level, before they become incidents.

Use Terraform Cloud or Sentinel for policy enforcement. A policy can require that all resources have a cost_center tag, that databases never allow public access, that encryption is enabled by default. Policies run on every plan, and non-compliant configurations are rejected:

import "tfplan/v2" as tfplan

deny_public_database = rule {
  all tfplan.resource_changes.aws_db_instance as db {
    db.change.after.publicly_accessible != true
  }
}

deny_untagged_resources = rule {
  all tfplan.resource_changes as resources {
    resources.change.after.tags contains "cost_center"
  }
}

main = rule {
  deny_public_database and deny_untagged_resources
}

Tag everything systematically. We use: environment (prod, staging, dev), cost_center (team owner), service (name of the service), managed_by (terraform, manually, etc.). These tags feed into billing reports, AWS Config, and cost optimization workflows.

Run cost estimation on every plan. Terraform Cloud and tools like Infracost show the monthly impact of a change before it's applied. A developer proposes a change, sees it costs $12k/month, and decides whether that's justified. This feedback loop prevents expensive surprises.

Scan Terraform code for security issues. Use tfsec, `checkov`, or similar tools in your CI pipeline. Catch hardcoded secrets, overly permissive IAM policies, and insecure defaults before they reach AWS. Treat security scan failures as pipeline blockers.

Common mistakes

Storing state locally in version control. Merge conflicts, leaks of sensitive data, and corrupted state are the trifecta. Always use remote state.

Monolithic state files. One state file for everything means one team can block another, and blast radius is unlimited. Split by component.

Applying manually from laptops. This breaks auditability, reproducibility, and introduces human error at scale. Automation is non-negotiable in production.

Forgetting to encrypt sensitive values. Passwords and API keys visible in state files are data breaches waiting to happen. Use sensitive = true and encrypt the state backend.

Building modules without documentation. A module is only valuable if the next person using it understands it. Invest in clear README files and example usage.

Skipping policy checks because they're inconvenient. Policies feel restrictive until they prevent your first incident. Then they're worth their weight in gold.

Key takeaways

• Remote state with encryption and locking is non-negotiable for production. Split state by component to reduce bottlenecks and blast radius.
• Modules must be small, versioned, and documented. They're how you scale infrastructure patterns across a team.
• Automate the entire workflow: plan in PRs, apply from CI using OIDC, audit every change. No manual applies in production.
• Policies, cost checks, and security scans prevent mistakes before they become incidents. Implement them early.
• Tag everything consistently. Tags are how you track costs, ownership, and lifecycle across thousands of resources.
• Treat state recovery and backup procedures as seriously as application backups. Document them and test recovery regularly.