Terraform Cloud and HCP: real-world lessons and examples

Most teams running Terraform at scale hit a wall: local state files become a bottleneck, enforcement becomes manual, and CI/CD pipelines need tighter controls. Terraform Cloud and HCP Terraform address these pain points head-on, but the gap between "it works" and "it works well in production" is wider than many expect. This article covers Terraform Cloud and HCP: real-world lessons and examples drawn from production deployments in e-commerce, SaaS, and fintech environments where teams have had to make hard choices about runs, state, policy, and module governance.

Remote state and locking included

Moving state from your laptop to Terraform Cloud is the first step most teams take, and for good reason. Remote state solves the human coordination problem: you no longer need a shared S3 bucket with manual locking scripts, and you avoid the situation where two engineers simultaneously run terraform apply against the same infrastructure.

Terraform Cloud's state management is automatic. Every workspace gets its own state file, versioning is built in, and locking happens transparently during runs. No more terraform force-unlock hacks at 2 AM. But here's what we learned the hard way: remote state is only half the story. You also need to think about state sensitivity.

State files contain secrets—database passwords, API keys, certificate data. Terraform Cloud encrypts state at rest and in transit, but you still need to:

• Restrict who can read state (use team permissions, not just workspace access).
• Audit state reads and writes (enable detailed logging in your Terraform Cloud audit trail).
• Rotate secrets regularly, even if they're encrypted in state.

Here's how we structure workspace organization for a typical SaaS deployment:

resource "tfe_workspace" "production" {
  name           = "prod-us-east-1"
  organization   = "acme-corp"
  terraform_version = "1.6.0"

  # Remote state enabled by default
  # Locking is automatic during runs
  
  # Restrict state access to prod-engineers team
  tag_names = ["production", "us-east-1"]
}

resource "tfe_team_access" "prod_engineers" {
  team_id      = tfe_team.prod_engineers.id
  workspace_id = tfe_workspace.production.id
  access       = "write"  # Can queue runs
}

resource "tfe_team_access" "devops" {
  team_id      = tfe_team.devops.id
  workspace_id = tfe_workspace.production.id
  access       = "read"   # Read-only audit access
}

One production lesson: state locks have timeouts. If a run hangs or crashes, Terraform Cloud holds the lock for a set period. We configure workspace timeout settings and always monitor for stuck runs via webhooks or the API.

Policy checks with Sentinel

Sentinel is Terraform Cloud's policy engine, and it's the difference between "we have Terraform" and "we have Terraform governance." Policies run after terraform plan but before terraform apply, allowing you to enforce organizational rules without code review.

In a fintech environment, we use Sentinel to enforce:

• Encryption at rest: All RDS instances must have storage_encrypted = true.
• Network isolation: Security groups must explicitly deny broad CIDR ranges.
• Cost guardrails: Reject instance types that exceed spending thresholds.
• Tagging compliance: All resources must have Environment and Owner tags.

Here's a real Sentinel policy that enforces encryption on S3 buckets and EBS volumes:

import "tfplan/v2" as tfplan

# Find all S3 buckets and EBS volumes
s3_buckets = filter tfplan.resource_changes as rc {
  rc.type == "aws_s3_bucket_server_side_encryption_configuration" and
  rc.mode == "managed" and
  (rc.change.actions contains "create" or rc.change.actions contains "update")
}

ebs_volumes = filter tfplan.resource_changes as rc {
  rc.type == "aws_ebs_volume" and
  rc.mode == "managed" and
  (rc.change.actions contains "create" or rc.change.actions contains "update")
}

# Validate S3 has encryption
s3_encrypted = rule {
  all s3_buckets as bucket {
    bucket.change.after.rule != null and
    bucket.change.after.rule[0].apply_server_side_encryption_by_default != null
  }
}

# Validate EBS has encryption
ebs_encrypted = rule {
  all ebs_volumes as vol {
    vol.change.after.encrypted == true
  }
}

main = rule {
  s3_encrypted and ebs_encrypted
}

Critical insight: policies can be too strict or too lenient. We've seen teams block all new infrastructure because they wrote a policy wrong, and we've seen teams disable policies because they were too annoying. The solution is iteration. Start permissive, log policy violations, refine based on actual data.

Also, Sentinel operates on the plan, not on the configuration. That means you're checking what will change, not just what you wrote. This catches edge cases where defaults don't match your requirements.

Private module registry

As your infrastructure code grows, you'll write the same pattern dozens of times: "create a VPC with subnets, security groups, and NAT gateways" or "launch a Kubernetes cluster with logging and monitoring." Terraform Cloud's private module registry lets you version and reuse these patterns across teams.

Instead of copy-pasting code or maintaining a shared GitHub repo everyone hacks on, you publish modules to your organization's registry and pin versions. This is how we've scaled Terraform across 50+ infrastructure engineers without forking code.

Publishing a module looks like this:

resource "tfe_registry_module" "vpc_base" {
  organization = "acme-corp"
  namespace    = "acme-corp"
  name         = "vpc"
  provider     = "aws"
  registry_name = "private"
  
  # Points to a GitHub repo containing the module
  # GitHub must have the naming convention: terraform-PROVIDER-NAME
}

In your workspace configuration, you consume it:

module "vpc" {
  source = "app.terraform.io/acme-corp/vpc/aws"
  version = "2.1.0"
  
  region = "us-east-1"
  cidr_block = "10.0.0.0/16"
  environment = "production"
}

We've found that private modules work best when:

• They're small and focused (one VPC, one EKS cluster, one RDS instance).
• They have clear versioning and a CHANGELOG.
• They enforce your company's security and compliance posture as defaults.
• They're documented with examples in the README.

Common pitfall: teams build one giant "company module" that does everything. It becomes unmaintainable and nobody knows which version they're using. Small, composable modules win.

VCS-driven workflows

The real power of Terraform Cloud emerges when you connect it to Git. Every push to a branch triggers a plan automatically. Every pull request shows the plan diff as a comment on GitHub. Merges to main trigger applies. This is the infrastructure-as-code workflow most teams aspire to.

Setup involves:

1. Connecting your Git organization (GitHub, GitLab, Bitbucket).
2. Creating a workspace linked to a specific repository and branch.
3. Configuring auto-apply (or requiring manual approval before apply).

Here's how we configure a VCS-driven workspace via Terraform:

resource "tfe_workspace" "infrastructure" {
  name              = "infrastructure-prod"
  organization      = "acme-corp"
  terraform_version = "1.6.0"
  
  # Link to VCS
  vcs_repo {
    identifier         = "acme-corp/infrastructure"
    branch             = "main"
    oauth_token_id     = tfe_oauth_client.github.oauth_token_id
  }
  
  # Auto-apply on main; require approval on PRs
  auto_apply = true
  
  # Queue runs only if files in this path change
  file_triggers_enabled = true
  working_directory     = "terraform/production"
}

resource "tfe_run_trigger" "on_module_change" {
  workspace_id  = tfe_workspace.infrastructure.id
  sourceable_id = tfe_workspace.terraform_modules.id
  
  # Rerun this workspace if modules update
}

Real-world gotchas:

• Working directory: If your repo has multiple Terraform stacks, use working_directory to avoid redundant plans. We structure repos as /terraform/production, /terraform/staging, etc.
• Speculative plans: Plans run on PRs are read-only and don't lock state. This is intentional, but it means you won't catch state conflicts until merge.
• Sensitive variables in Git: Never commit secrets. Use Terraform Cloud variables and mark them sensitive so they're masked in logs.
• Branch protection: Enforce approvals on pull requests and require status checks to pass before merge. We require at least two team members to review infrastructure changes.

One team we worked with had every engineer push to main directly. They switched to VCS-driven workflows and immediately caught a bug where someone was creating unencrypted RDS instances. The plan showed up as a PR comment, another engineer said "wait, why isn't encryption enabled?" and the issue was fixed before merge. That's the friction you want.

Common mistakes

• Mixing remote and local state: If you have some workspaces in Terraform Cloud and others running locally, you'll eventually have state drift and confusion. Commit to one approach.
• Not versioning Terraform itself: Set terraform_version in your workspace to pin the version. Upgrading Terraform can change plan output; do it deliberately, not accidentally.
• Weak variable permissions: If your Terraform Cloud workspace can read production secrets, every developer with workspace access can read them. Use narrower permissions and rotate often.
• Ignoring run logs: When something breaks, the logs are in Terraform Cloud. Most teams don't check them; they blame "the cloud" instead of reading what actually happened.
• Over-automating apply: Auto-apply is convenient but dangerous. We use it only for lower environments. Production changes require explicit approval and a waiting period so people can object.

Key takeaways

• Remote state eliminates coordination problems, but you still need to secure and audit it like any other sensitive data.
• Sentinel policies should start permissive and evolve based on violations you actually see, not hypothetical risks.
• Private modules scale governance but only if they're small, well-versioned, and focused on one pattern each.
• VCS-driven workflows enforce audit trails and peer review; branch protection and speculative plans catch mistakes before they reach production.
• Pin Terraform versions in workspaces, use narrow permissions, and always read logs when something fails.