Drift detection: Terraform & IaC lessons from production

Drift detection: Terraform & IaC lessons from production

We've all been there: your Terraform state says the security group has three rules, but the console shows five. A colleague manually toggled a Lambda's timeout through the AWS UI to fix a "quick" production issue, and now your infrastructure code no longer matches reality. Drift detection—the practice of identifying when actual cloud resources diverge from your infrastructure-as-code definitions—is often treated as an afterthought, but teams running production systems at scale find it's essential to catch configuration drift before it cascades into incidents.

Why drift happens in the real world

Infrastructure drift isn't a theoretical problem. In our experience across e-commerce platforms, trading systems, and SaaS deployments, drift creeps in through several common channels:

• Emergency fixes. A database parameter group is modified during an incident to restore latency; the fix goes live, but the Terraform code is updated weeks later—or never.
• Third-party integrations. A monitoring tool auto-scales an Auto Scaling Group, or a security scanning service modifies S3 bucket policies.
• Accidental console changes. A junior engineer explores the console and unknowingly changes a resource tag or security group rule.
• Manual rollbacks. A deployment failed, and someone reverted the stack using the console rather than `terraform apply` to roll back the code.

The risk is that your infrastructure-as-code becomes a lie. When you deploy the next change, Terraform might overwrite the undocumented production fix, causing an outage. Or worse, security auditors find misconfigurations that your code never intended.

Run terraform plan on a schedule

The foundation of drift detection is automated, scheduled drift checks. Rather than waiting for the next deployment to discover mismatches, we run terraform plan periodically—typically every 12 or 24 hours—and compare the planned changes to what's actually deployed. If terraform plan reports changes when nothing was intentionally modified, you've found drift.

Here's a practical setup using GitHub Actions (the pattern works equally well with GitLab CI, AWS CodePipeline, or Jenkins):

name: Scheduled Drift Detection
on:
  schedule:
    - cron: '0 2 * * *'  # 02:00 UTC daily
  workflow_dispatch:

jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.7.0
      
      - name: Terraform Init
        run: terraform init
        env:
          AWS_REGION: us-east-1
          AWS_ACCESS_KEY_ID: ${{ secrets.TF_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.TF_AWS_SECRET_ACCESS_KEY }}
      
      - name: Terraform Plan (Drift Check)
        id: plan
        run: terraform plan -no-color -detailed-exitcode
        env:
          AWS_REGION: us-east-1
          AWS_ACCESS_KEY_ID: ${{ secrets.TF_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.TF_AWS_SECRET_ACCESS_KEY }}
        continue-on-error: true
      
      - name: Comment on Drift
        if: steps.plan.outputs.exitcode == 2
        run: |
          echo "⚠️ Drift detected! Infrastructure has diverged from code."
          terraform plan -no-color

The exit code from terraform plan tells us whether drift exists: exit code 0 means no changes, exit code 2 means changes detected (drift). By running this on a schedule, you catch divergence in hours or days rather than at the next deployment.

Alert on unexpected changes

Detecting drift is only half the battle; you need to be notified when it happens. In production environments, we route drift alerts to your on-call team immediately so someone can investigate while the context is fresh.

Extend the GitHub Actions workflow to send alerts to Slack, PagerDuty, or your monitoring system:

      - name: Notify Slack on Drift
        if: steps.plan.outputs.exitcode == 2
        uses: slackapi/slack-github-action@v1.24.0
        with:
          webhook-url: ${{ secrets.SLACK_WEBHOOK }}
          payload: |
            {
              "text": "🚨 Infrastructure Drift Detected",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "*Drift Detected in Production*\nEnvironment: prod\nTime: ${{ github.event.scheduled_time }}\nCheck the full plan in the workflow run."
                  }
                },
                {
                  "type": "actions",
                  "elements": [
                    {
                      "type": "button",
                      "text": {
                        "type": "plain_text",
                        "text": "View Workflow"
                      },
                      "url": "${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
                    }
                  ]
                }
              ]
            }

Some teams also write the drift output to a centralized log (CloudWatch, DataDog, Splunk) and set threshold-based alerts: if drift is detected more than twice in a week, escalate to engineering leadership.

Find who changed what in CloudTrail

When drift is detected, the first question is always: who made this change and when? AWS CloudTrail records every API call, including manual console changes. It's your audit trail.

Set up CloudTrail to log all API events to an S3 bucket (if not already done), then query it when drift is found. Here's a practical approach using AWS CLI and jq:

#!/bin/bash
# Query CloudTrail for changes to a specific resource
# Usage: ./find-drift-source.sh sg-12345678

RESOURCE_ID=$1
START_TIME=$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%S)Z
END_TIME=$(date -u +%Y-%m-%dT%H:%M:%S)Z

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=$RESOURCE_ID \
  --start-time $START_TIME \
  --end-time $END_TIME \
  --max-results 50 \
  --region us-east-1 | jq '.Events[] | {
    EventTime: .EventTime,
    Username: .Username,
    EventName: .EventName,
    SourceIPAddress: .CloudTrailEvent | fromjson | .sourceIPAddress
  }'

This query returns the event time, the IAM user or role that made the change, the API call type (e.g., AuthorizeSecurityGroupIngress), and the source IP. In practice, you'll find that most console drift comes from specific users or IP ranges, making it easy to identify whether it was an accident, an undocumented fix, or a security incident.

For teams with detailed audit requirements, consider storing CloudTrail logs in a data lake and querying them with Athena or creating dashboards in your observability tool that show drift sources over time.

Remove console write access for managed resources

The strongest drift prevention is removing the ability to make changes through the console in the first place. Teams we've worked with use resource-based IAM policies or service control policies (SCPs) to prevent direct modification of infrastructure managed by Terraform.

Here's an SCP that blocks console changes to EC2 security groups in production:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyDirectSecurityGroupModification",
      "Effect": "Deny",
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:ModifySecurityGroupRules"
      ],
      "Resource": "arn:aws:ec2:*:ACCOUNT_ID:security-group/*",
      "Condition": {
        "StringNotEquals": {
          "aws:PrincipalArn": [
            "arn:aws:iam::ACCOUNT_ID:role/terraform-runner",
            "arn:aws:iam::ACCOUNT_ID:role/emergency-access"
          ]
        }
      }
    }
  ]
}

This policy allows only the Terraform service role and a dedicated emergency-access role to modify security groups; everyone else is blocked. You can apply this pattern to RDS parameter groups, Lambda configurations, S3 bucket policies, and any other resource managed by Terraform.

The trade-off is that fixing production issues now requires code changes and redeployment (or using the emergency-access role with full audit logging). Teams often allow the emergency role as an escape hatch for true incidents, but that role is monitored closely and requires peer review of changes.

Common mistakes

• Running drift checks but not acting on them. If you ignore drift alerts consistently, the noise becomes invisible. Treat drift detection alerts with the same urgency as security findings.
• Forgetting to re-apply after identifying drift. Finding drift is only half the job; you need a clear process for deciding whether to revert the manual changes or merge them into Terraform, then actually execute that decision.
• Assuming drift only happens in production. Drift in staging or development environments teaches you how changes leak into production. Watch all environments.
• Not excluding managed tags or read-only attributes. Use lifecycle { ignore_changes = [tags, ...] } to prevent false positives from tags added by AWS services or third-party tools.
• Relying on drift detection as the only safeguard. Drift detection catches the problem; IAM policies prevent it. Use both together.

Key takeaways

• Schedule terraform plan daily (or more frequently) to catch drift early, before it causes outages.
• Alert your team immediately when drift is detected; timely investigation prevents cascading failures.
• Query CloudTrail to identify who made undocumented changes and whether they were intentional or accidental.
• Use IAM policies and SCPs to block console access to resources managed by Terraform, reducing drift at the source.
• Combine drift detection (reactive) with access controls (proactive) for a robust infrastructure governance strategy.
• Treat drift resolution as a normal operational task; update your Terraform code to reflect intentional changes or revert unintended ones.