Secrets in Terraform in Terraform, IaC & Cloud Automation

The Cost of a Leaked Database Password

We've all heard the horror stories: a developer accidentally commits AWS credentials to a public repository, a malicious actor finds them within hours, and suddenly your production database is being ransomed or wiped. When we talk about secrets in Terraform in Terraform, IaC & Cloud Automation, we're really talking about preventing exactly this scenario. The challenge is that infrastructure-as-code makes it easier than ever to manage resources at scale, but it also makes it easier than ever to accidentally hardcode a password, API key, or certificate into version control.

Over 40 days of building infrastructure automation, you've learned how to modularize, scale, and standardize your cloud deployments. But none of that matters if your secrets end up in a Git log or stored plaintext in Terraform state. This article walks through the patterns and tools we use in production to keep secrets safe while maintaining the automation benefits that IaC provides.

Never Commit Secrets to tfvars

The most dangerous trap in Terraform is the terraform.tfvars file. It's convenient: you define a variable for your database password, set it in a .tfvars file, and forget about it. Then, when you commit your code to Git, that file goes right into history alongside your main.tf.

Even if you add terraform.tfvars to .gitignore after the fact, it's already in the repository history. Any team member who clones the repo can check out an earlier commit and read every secret ever stored there. Tools like git-secrets or GitHub's secret scanning can help catch this, but detection is a bandage on a deeper problem.

Instead of storing secrets in .tfvars files, we generate them in AWS Secrets Manager or Parameter Store before Terraform runs. This way, Terraform references the secret by its ARN or name, not by its actual value.

# Bad: Never do this
variable "db_password" {
  type        = string
  description = "Database password"
  default     = "super-secret-123"  # Exposed in Git history
}

resource "aws_db_instance" "main" {
  engine               = "postgres"
  engine_version       = "14.7"
  master_username      = "postgres"
  master_password      = var.db_password  # Plaintext in state
  allocated_storage    = 20
  skip_final_snapshot  = false
}

The master_password value ends up plaintext in the Terraform state file. Anyone with access to state (or your S3 backend) can read it. This is the second risk: Terraform state is not encrypted by default.

Generate Secrets in AWS Secrets Manager

AWS Secrets Manager is the right place to create and rotate secrets outside of your Terraform workflow. It integrates with AWS Lambda, RDS, and other services for automatic rotation. The key insight is that Secrets Manager is the source of truth for the actual secret value, not Terraform.

In production, we usually have a separate "secrets" stack that runs once to initialize Secrets Manager entries. This stack can be deployed by a CI/CD pipeline with restricted IAM permissions, or even manually by an operator. After that, Terraform only references these secrets, never creates or modifies them.

Here's how we generate a database password in Secrets Manager first:

# In a separate, restricted stack
resource "aws_secretsmanager_secret" "db_password" {
  name                    = "prod/rds/postgres/master_password"
  recovery_window_in_days = 7
  description             = "RDS master password for production database"
}

resource "aws_secretsmanager_secret_version" "db_password" {
  secret_id = aws_secretsmanager_secret.db_password.id
  # Generate a random password (or supply your own)
  secret_string = random_password.db.result
}

resource "random_password" "db" {
  length  = 32
  special = true
}

Once this secret exists in Secrets Manager, the main Terraform stack uses a data source to retrieve only the ARN, not the value itself:

# In your main application stack
data "aws_secretsmanager_secret" "db_password" {
  name = "prod/rds/postgres/master_password"
}

# Now retrieve the actual secret version for RDS
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = data.aws_secretsmanager_secret.db_password.id
}

resource "aws_db_instance" "main" {
  engine               = "postgres"
  engine_version       = "14.7"
  master_username      = "postgres"
  master_password      = jsondecode(data.aws_secretsmanager_secret_version.db_password.secret_string)["password"]
  allocated_storage    = 20
  skip_final_snapshot  = false
  db_name              = "production"
  # Enable encryption at rest
  storage_encrypted    = true
  kms_key_id           = aws_kms_key.rds.arn
}

Notice that we only reference the secret via the data source. The actual password value is passed directly to RDS and does not linger in Terraform state (though it is present during the resource creation).

Reference ARNs, Not Values

A core principle in securing infrastructure code: whenever possible, reference resources by their ARN or name, not by their value. This applies to secrets, keys, certificates, and any sensitive data.

When your Terraform configuration outputs an ARN, it's safe to commit that to version control. An ARN is a unique identifier; knowing the ARN of a secret doesn't give an attacker the secret itself. But the value of a secret can be retrieved by anyone with the right IAM permissions and access to Secrets Manager.

This pattern extends beyond Secrets Manager:

• TLS certificates: Reference the ARN from AWS Certificate Manager, not the certificate body.
• KMS keys: Use the key ARN to enable encryption, not the key material.
• SSH keys: Store the public key in Parameter Store, not the private key.
• API tokens: Keep them in Secrets Manager and reference by name.

In your outputs, be equally careful. Never output a secret value, even if it's "only for the team." Instead, output the Secrets Manager name or ARN:

# Bad
output "db_password" {
  value       = data.aws_secretsmanager_secret_version.db_password.secret_string
  description = "Database password"
}

# Good
output "db_password_secret_arn" {
  value       = data.aws_secretsmanager_secret.db_password.arn
  description = "ARN of the database password secret"
}

Team members who need the actual password can retrieve it via the AWS Console or the AWS CLI using their own IAM credentials. This way, we maintain an audit trail of who accessed the secret and when.

Remember: State Stores Values in Plain Text

This bears repeating because it catches so many teams off guard: Terraform state files store resource attributes in plain text by default. If a resource's password, API key, or token is stored in state, anyone with read access to state can extract it.

We use two layers of protection:

1. Encrypt state in transit and at rest: Store state in S3 with server-side encryption (SSE-S3 or SSE-KMS) and enable versioning and MFA delete.
2. Restrict state access: Use IAM policies to limit who can read or write the state bucket. Only CI/CD pipelines and trusted operators should have access.

Even with encryption, the rule is simple: do not let sensitive values end up in state in the first place. When you use Secrets Manager the way we've shown, the actual password is created and stored there, not in Terraform. Terraform only knows about the secret's ARN or name.

# Encrypt Terraform state in S3
resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-org-terraform-state-prod"
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.terraform_state.arn
    }
  }
}

# Block public access
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

Additionally, enable logging on your state bucket and monitor CloudTrail for unauthorized access attempts. If someone does manage to breach the state file, you want to know about it immediately.

Common Mistakes

Mistake 1: Checking in example tfvars with real secrets. An "example" file is still a file that gets committed. Use a dedicated example.tfvars that contains only dummy values (e.g., password = "changeme"). Make it clear in your README that no real secrets should ever appear in version control.

Mistake 2: Rotating secrets without updating Terraform. If your operator rotates a password in Secrets Manager but Terraform doesn't know about it, the next terraform apply might revert the rotation. Document the contract clearly: Terraform creates the secret name and retrieves the current version, but operators own the password rotation policy.

Mistake 3: Outputting secret ARNs in logs. Even an ARN can leak if your logs are verbose. Use sensitive = true in outputs to prevent Terraform from printing values in logs and state.

Mistake 4: Using default VPC or no encryption. A compromised secret is only dangerous if it can be used. Combine secrets management with network isolation (VPC, security groups) and encryption at rest. Defense in depth applies here too.

Key Takeaways

• Never store secrets in .tfvars files or any file committed to version control. Use AWS Secrets Manager as the authoritative source for secret values.
• Reference secrets by ARN or name in Terraform, never by value. Outputs should point to the secret location, not its contents.
• Encrypt Terraform state in S3 using KMS and restrict IAM access to the state bucket. State stores attribute values in plaintext, so defense at the storage layer is critical.
• Implement automatic secret rotation in Secrets Manager and audit who accesses secrets via CloudTrail. Separate the secret creation (a restricted operation) from the secret retrieval (what Terraform does).
• Use sensitive = true on outputs and resource arguments that contain secrets to prevent Terraform from printing them in logs.
• Combine secrets management with network isolation and encryption at rest. No single control is sufficient; use defense in depth.