Blue/green deployments on AWS: best practices explained

Production deployments carry risk. A single misconfiguration or unexpected behavior can cascade into downtime across your infrastructure. Blue/green deployments on AWS offer a proven pattern to eliminate that uncertainty: you run two identical environments in parallel, validate the new one completely, then switch traffic with a single command—and roll back just as fast if something goes wrong.

We've used this approach extensively across our e-commerce and SaaS stacks, and the peace of mind it brings to deployment day is worth every hour spent setting it up. Let's walk through the mechanics, best practices, and gotchas we've learned along the way.

CodeDeploy supports ECS blue/green natively

AWS CodeDeploy is purpose-built for blue/green deployments. If you're running containers on ECS, CodeDeploy handles the orchestration: it spins up your new (green) environment, validates it's healthy, and then either shifts traffic or rolls back—all without manual intervention.

The integration works like this: CodeDeploy watches your ECS service and Application Load Balancer (ALB). When you trigger a deployment, it creates a new task set (the green environment) alongside your existing one (blue). Both run simultaneously. CodeDeploy monitors CloudWatch alarms and target group health checks to determine if the green environment is ready. Once it passes validation, CodeDeploy updates the ALB to route traffic to the new tasks.

Here's a minimal ECS service definition that supports blue/green deployments:

{
  "service": "my-api-service",
  "cluster": "production",
  "taskDefinition": "my-api:42",
  "desiredCount": 3,
  "deploymentController": {
    "type": "CODE_DEPLOY"
  },
  "loadBalancers": [
    {
      "containerName": "api",
      "containerPort": 8080,
      "targetGroupArn": "arn:aws:elasticloadbalancing:..."
    }
  ]
}

The critical piece is deploymentController.type set to CODE_DEPLOY. This tells ECS to let CodeDeploy manage the traffic shifting, not ECS's built-in rolling deployment strategy.

Your CodeDeploy application configuration (appspec.yaml) then defines how the deployment should behave:

version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: !Ref TaskDefinitionArn
        LoadBalancerInfo:
          ContainerName: api
          ContainerPort: 8080
        PlatformVersion: LATEST
        NetworkConfiguration:
          AwsVpcConfiguration:
            Subnets:
              - subnet-12345678
            SecurityGroups:
              - sg-12345678
            AssignPublicIp: ENABLED

Hooks:
  - BeforeAllowTraffic: "validate-deployment"
  - AfterAllowTraffic: "smoke-tests"

Notice the Hooks section. This is where you tell CodeDeploy to run validation logic before and after the traffic shift. Those hooks invoke Lambda functions that you define—we'll cover that in the next section.

Test the green environment before shifting

Here's where most teams stumble: they assume CodeDeploy will automatically detect broken deployments. It won't, unless you tell it what to check for. CodeDeploy only looks at the metrics and alarms you specify. If your code is syntactically valid but logically broken, it will happily route 100% of production traffic to it.

The BeforeAllowTraffic hook is your safety net. Before CodeDeploy touches the load balancer, you run a Lambda function that exercises the green environment: hit key endpoints, verify database connectivity, check cache population, test critical workflows. Only if all checks pass does CodeDeploy proceed to shift traffic.

Here's a practical validation Lambda function:

import json
import boto3
import requests
import time

codedeploy = boto3.client('codedeploy')
elbv2 = boto3.client('elbv2')

def lambda_handler(event, context):
    deployment_id = event['DeploymentId']
    lifecycle_event_hook_id = event['LifecycleEventHookExecutionId']
    
    try:
        # Get the target group for the green environment
        tg_response = elbv2.describe_target_groups(
            Names=['my-api-green']
        )
        tg_arn = tg_response['TargetGroups'][0]['TargetGroupArn']
        
        # Get the green targets (newly deployed instances)
        targets = elbv2.describe_target_health(TargetGroupArn=tg_arn)
        
        if not targets['TargetHealthDescriptions']:
            raise Exception("No healthy targets in green environment")
        
        # Wait for targets to stabilize
        time.sleep(5)
        
        # Run validation checks
        for target in targets['TargetHealthDescriptions']:
            if target['TargetHealth']['State'] != 'healthy':
                raise Exception(f"Target {target['Target']['Id']} is not healthy")
            
            # Hit the health endpoint
            ip = target['Target']['Id']
            response = requests.get(f'http://{ip}:8080/health', timeout=5)
            
            if response.status_code != 200:
                raise Exception(f"Health check failed on {ip}")
            
            health_data = response.json()
            
            # Verify database connectivity
            if not health_data.get('database_connected'):
                raise Exception("Database connection check failed")
            
            # Verify cache is warm
            if not health_data.get('cache_ready'):
                raise Exception("Cache not ready")
        
        # All checks passed
        codedeploy.put_lifecycle_event_hook_execution_status(
            deploymentId=deployment_id,
            lifecycleEventHookExecutionId=lifecycle_event_hook_id,
            status='Succeeded'
        )
        
    except Exception as e:
        print(f"Validation failed: {str(e)}")
        codedeploy.put_lifecycle_event_hook_execution_status(
            deploymentId=deployment_id,
            lifecycleEventHookExecutionId=lifecycle_event_hook_id,
            status='Failed'
        )
        raise

This Lambda checks that all green targets are healthy, verifies critical endpoints respond correctly, and confirms dependencies like databases and caches are reachable. If any check fails, CodeDeploy stops the deployment immediately and keeps traffic routed to blue. No production impact.

Shift gradually with canary settings

Once validation passes, you have options for how to shift traffic. The aggressive approach—shift 100% immediately—works fine, but canary shifts give you an additional safety valve: you send a small percentage of traffic to green first, monitor it for errors and latency spikes, then gradually increase the percentage.

Configure canary shifts in your CodeDeploy deployment group:

aws deploy create-deployment-group \
  --application-name my-api \
  --deployment-group-name prod-deployment \
  --service-role-arn arn:aws:iam::123456789012:role/CodeDeployRole \
  --deployment-config-name CodeDeployDefault.ECSCanary10Percent5Minutes \
  --load-balancer-info targetGroupName=my-api-tg

The deployment config CodeDeployDefault.ECSCanary10Percent5Minutes means: shift 10% of traffic to green, wait 5 minutes, then shift the remaining 90%. CodeDeploy watches CloudWatch alarms you've configured during those 5 minutes. If any alarm triggers (high error rate, elevated latency, custom metrics), CodeDeploy automatically rolls back to blue.

You might also use CodeDeployDefault.ECSLinear10PercentEvery10Minutes, which increases traffic by 10% every 10 minutes. This spreads the risk across a longer window, giving you more time to catch issues:

Wire up CloudWatch alarms that CodeDeploy will monitor:

aws cloudwatch put-metric-alarm \
  --alarm-name my-api-green-error-rate \
  --alarm-description "High error rate on green environment" \
  --metric-name ErrorRate \
  --namespace AWS/ApplicationELB \
  --statistic Average \
  --period 60 \
  --threshold 5 \
  --comparison-operator GreaterThanThreshold \
  --dimensions Name=TargetGroup,Value=targetgroup/my-api-green/* \
  --evaluation-periods 2

If error rate exceeds 5% for 2 consecutive 60-second periods, the alarm fires and CodeDeploy stops the traffic shift. Blue remains active, and your users never see the problem.

Keep blue running until confident

A common optimization teams try is terminating the blue environment as soon as green takes traffic. Don't. Keep blue running for at least 15–30 minutes after the traffic shift completes. This gives you a rollback target if problems surface after canary validation passes but before full traffic migration.

Here's why this matters: canary monitoring catches obvious failures (500 errors, timeouts). But some issues only emerge under full production load—subtle database connection pool exhaustion, memory leaks that compound over hours, cache coherency issues that take time to manifest. Keeping blue running costs you the price of one extra task set for a short window. The insurance is worth it.

CodeDeploy has a built-in termination delay. Configure it to keep blue running for a defined period:

aws deploy create-deployment-group \
  --application-name my-api \
  --deployment-group-name prod-deployment \
  --deployment-config-name CodeDeployDefault.ECSCanary10Percent5Minutes \
  --blue-green-deployment-configuration '{
    "terminateBlueInstancesOnDeploymentSuccess": {
      "action": "TERMINATE",
      "terminationWaitTimeInMinutes": 30
    }
  }'

Set terminationWaitTimeInMinutes to 30. After the traffic shift completes and 30 minutes elapse without triggering a rollback, CodeDeploy terminates the blue environment. If you need to roll back during that window, CodeDeploy shifts traffic back to blue, which is still fully running and warm.

If issues do surface during that 30-minute window, rolling back is a single API call:

aws deploy stop-deployment \
  --deployment-id d-ABC123DEF456 \
  --auto-rollback-enabled

Traffic returns to blue within seconds. No customer impact.

Common mistakes

Not validating the green environment: Teams configure CodeDeploy but skip the validation hooks. CodeDeploy will happily deploy broken code. Always run the BeforeAllowTraffic hook with meaningful checks.

Relying solely on ALB health checks: ALB health checks verify that the container is running and responding on a health endpoint. They don't verify that your application logic is correct. A container can be "healthy" while returning stale or incorrect data. Implement application-level validation in your Lambda hooks.

Setting CloudWatch alarms with poor thresholds: If your error rate alarm threshold is too high, it won't catch real problems. If it's too low, you'll get false positives and rollbacks during expected spikes. Spend time baselining your metrics. An error rate of 0.1% might be normal for your service; set the alarm at 1% to catch genuine issues.

Terminating blue too quickly: Saving on compute costs by terminating blue immediately is false economy. You lose your rollback target and accept risk. Keep blue running for at least 15 minutes.

Not testing the rollback procedure: Practice rolling back a deployment in staging before you need to do it in production. Run through the full flow, including verifying that blue is still healthy and capable of handling traffic after being dormant.

Key takeaways

• CodeDeploy with ECS handles blue/green orchestration. Set deploymentController.type to CODE_DEPLOY in your ECS service definition.
• Use the BeforeAllowTraffic hook to validate the green environment before traffic shifts. Run health checks, dependency checks, and critical workflows from a Lambda function.
• Implement canary or linear traffic shifting. Start with a small percentage of traffic, monitor CloudWatch alarms for 5–10 minutes, then complete the shift. This catches issues before they affect all users.
• Keep the blue environment running for 15–30 minutes after traffic shifts. This provides a warm rollback target if problems emerge after validation.
• Set up CloudWatch alarms for error rate, latency, and business metrics. CodeDeploy monitors these during the traffic shift and rolls back automatically if thresholds are breached.
• Practice rolling back in staging. Test the full blue/green cycle before relying on it in production.