State locking explained: real-world lessons and examples

Opening

You're halfway through a Friday afternoon when Slack lights up: your colleague's terraform apply is hanging, and your deploy pipeline is queued behind it. Two team members accidentally triggered infrastructure changes at the same time, and now you're staring at a corrupted state file and a rollback nightmare. State locking explained: real-world lessons and examples is not just theory—it's the difference between a smooth deployment and a Saturday incident response. Terraform state locking is your guard rail against concurrent writes that silently corrupt infrastructure metadata, leaving your environment in an undefined state.

Why locks stop concurrent writes

Terraform state is a source of truth. It maps your resource definitions to real cloud objects. When two terraform apply or terraform plan operations run simultaneously on the same state file, they can both read the same state version, make conflicting changes, and write back different versions. The last write wins—but the losing write's changes are lost entirely, and your infrastructure drifts into inconsistency.

State locking introduces a mutex. Before Terraform reads or writes state, it acquires a lock. If another operation holds the lock, the new operation blocks and waits. Once the lock is released, the next operation proceeds. This enforces strict serialization of state mutations.

We learned this the hard way in a SaaS environment. A developer manually ran terraform apply while CI was already applying. The manual run read state, made changes to a database security group rule, wrote back. CI read the old state, created an IAM role, wrote back. The security group changes were lost. The role was never created. Both resources appeared correct in Terraform code, but the actual cloud state was split between two versions of truth.

S3 native lockfile vs. DynamoDB

Terraform's locking mechanism differs depending on your backend. For local state, there is no distributed locking—only a file-level lock on your machine. For remote backends, you need distributed coordination.

S3 backend with DynamoDB

The de facto standard for AWS environments pairs S3 for state storage with DynamoDB for locking. S3 holds the state file; DynamoDB holds lock records. When Terraform acquires a lock, it writes an item to a DynamoDB table with a unique ID and timestamp. Other operations check for this item before proceeding.

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The DynamoDB table must have a primary key named LockID (string). Terraform creates and deletes lock items automatically:

resource "aws_dynamodb_table" "terraform_locks" {
  name           = "terraform-locks"
  billing_mode   = "PAY_PER_REQUEST"
  hash_key       = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }

  tags = {
    Name = "Terraform State Lock Table"
  }
}

In production, we run this setup in a separate "bootstrap" AWS account or a locked-down state management stack. The table is off-limits to regular developers; only Terraform's service role can read and write it.

Azure and GCP backends

Azure Blob Storage backend uses blob leases for locking. GCP Cloud Storage backend uses object-level metadata. Both are transparent—Terraform handles them automatically. The principle is identical: a distributed coordination mechanism prevents overlapping writes.

force-unlock only when absolutely certain

terraform force-unlock is the nuclear option. It deletes the lock without running the operation that created it. Use it only when you are 100% certain the operation holding the lock has genuinely crashed and will not resume.

A real scenario: a Terraform operator in our ride-hailing platform ran terraform apply from their laptop, then their machine crashed mid-apply. The lock remained in DynamoDB for hours. The operation was clearly dead. We verified no process was still running, checked CI logs to confirm no concurrent apply was in progress, then issued terraform force-unlock:

terraform force-unlock abc123def456

But consider the inverse: a CI build times out and appears to hang, so an operator force-unlocks the state. Thirty seconds later, the original CI job resumes, modifies state, and writes back. Now you have two operations scribbling on the state file, and the lock is useless. The state is corrupted.

Before force-unlocking, always:

• Verify the lock holder's process is dead (check CI logs, process managers, log aggregation).
• Confirm no other team member is actively running Terraform on that workspace.
• Review the lock's timestamp to gauge how stale it is.
• Notify the team in Slack or your incident channel.
• Document the unlock in your runbook or incident tracker.

Most important: adjust your process to prevent lock holders from crashing silently. Set timeouts in CI. Monitor for hung builds. Use Terraform's -lock-timeout flag to give operations a reasonable wait period before failing hard.

CI must serialize applies

State locking is a guard rail, not a replacement for process discipline. Your CI/CD pipeline must be architected to never launch two applies concurrently on the same workspace.

Pipeline concurrency controls

In a typical GitOps flow, a single queue enforces serialization. When a PR is merged to main, a CI job validates and applies Terraform. If another PR merges before the first apply completes, the second job waits in the queue. No two applies run simultaneously:

stages:
  - validate
  - plan
  - apply

apply:
  stage: apply
  script:
    - terraform init
    - terraform apply -auto-approve -lock-timeout=10m
  only:
    - main
  resource_group: prod-tf-apply  # Serializes this job

The resource_group (in GitLab) or equivalent (in GitHub Actions, use concurrency groups; in Jenkins, use build queues) ensures only one job with that label runs at a time. Other jobs wait.

Manual applies and CI conflict

The trickiest scenario: a developer manually runs terraform apply from their laptop while CI is also applying. State locking prevents corruption, but it also blocks CI. CI waits 10 minutes (the lock timeout), then fails. Now the team has to decide: was the manual apply legitimate, or was it accidental?

In a mature e-commerce operation we've worked with, manual applies are forbidden on production workspaces. All changes flow through CI, which has approval gates. Development and staging allow manual applies, but with explicit Slack notifications: a bot posts whenever someone acquires a state lock, and the team's Slack channel sees a 5-minute countdown. If someone force-unlocks without warning, the bot alerts the team.

Implement this using a custom wrapper script or Terraform cloud's policy engine. Better yet, enforce it via IAM: deny terraform:GetState and terraform:PutState for production workspaces except for your CI service role.

Common mistakes

• Misconfiguring the DynamoDB table. A common error is naming the primary key something other than LockID. Terraform will try to write a lock item and fail silently, leaving your state vulnerable to concurrent writes. Always verify the table schema matches Terraform's expectations.
• Forgetting to encrypt the state file. Locking protects against concurrent writes, but encryption protects against unauthorized reads. Enable S3 encryption (server-side AES-256 or KMS) in the same terraform block.
• Sharing workspaces across environments. A dev and a prod workspace should never share a state backend. Separate backends, separate DynamoDB tables, separate service roles. A lock on prod should never block a dev apply.
• Assuming force-unlock is safe because state was locked. Force-unlock tells Terraform the lock is gone, but it does not verify that the operation holding the lock is actually dead. Use it sparingly and defensively.
• Not setting lock timeouts in CI. Without a timeout, a hung Terraform process holds the lock forever. Set -lock-timeout=10m (or your SLA-appropriate value) so CI fails hard if it cannot acquire a lock within a reasonable time.

Key takeaways

• State locking is a distributed mutex that prevents concurrent reads and writes to Terraform state, protecting infrastructure consistency.
• S3 + DynamoDB is the standard AWS setup; other cloud providers have analogous mechanisms (blob leases for Azure, object metadata for GCP).
• DynamoDB locking requires a table with LockID as the primary key; misconfiguring it leaves state vulnerable.
• force-unlock should only be used after confirming the lock-holding process is genuinely dead; use it thoughtfully and document it.
• CI pipelines must enforce serialization (resource groups, concurrency limits, or build queues) to prevent concurrent applies, even though state locking provides a fallback.
• Set -lock-timeout in CI to fail fast if a lock cannot be acquired, preventing indefinite hangs.
• Separate state backends, DynamoDB tables, and service roles for each environment (dev, staging, prod) to avoid cross-environment lock collisions.
• Lock metadata (timestamp, ID, operation) is often logged; monitor these logs for patterns of frequent lock conflicts, which may indicate process issues.