Route 53 health checks and failover in AWS for Developers

Route 53 Health Checks and Failover

When your primary API endpoint goes down at 3 AM on a Friday, you need DNS to automatically route traffic to a healthy backup—that's where Route 53 health checks and failover come in. In this article, we'll explore how to set up automatic DNS failover between regions or endpoints, ensuring your application remains available even when individual instances fail. This is one of the most practical high-availability patterns I've implemented across e-commerce and SaaS platforms.

Understanding Health Checks

Health checks are the foundation of automatic failover. Route 53 can monitor your endpoints continuously, checking their health status at regular intervals. I typically think of health checks as a simple question: "Is this endpoint alive and responding correctly?" If the answer changes from yes to no, Route 53 can act on that information immediately.

Route 53 supports three types of health checks:

• HTTP/HTTPS health checks: Route 53 sends periodic requests to your endpoint and validates the response. You can check specific paths, expected status codes, and response strings.
• TCP health checks: Simple connection tests on a specific port without sending application-level data. Useful for databases or custom services.
• CloudWatch alarm-based health checks: Delegate health determination to CloudWatch alarms you've already configured, letting you use custom metrics and logic.

Here's a practical example: imagine running an e-commerce API in us-east-1 and us-west-2. You want Route 53 to detect if us-east-1 becomes unhealthy and automatically route new requests to us-west-2. That detection mechanism is a health check.

When I set up a health check for an HTTP endpoint, I typically use these parameters:

# Health check configuration (conceptual AWS CLI format)
aws route53 create-health-check \
  --health-check-config \
    IPAddress=203.0.113.10,\
    Port=443,\
    Type=HTTPS,\
    ResourcePath=/health,\
    FullyQualifiedDomainName=api.example.com,\
    RequestInterval=30,\
    FailureThreshold=3

The RequestInterval of 30 seconds means Route 53 sends a request every 30 seconds. The FailureThreshold of 3 means the endpoint must fail 3 consecutive checks before Route 53 marks it unhealthy. This prevents flaky networks from triggering unnecessary failovers.

Failover Routing Switches to a Backup

Failover routing is Route 53's way of saying, "If the primary is down, use the secondary." It's different from weighted routing or latency-based routing because it has an explicit primary-backup relationship.

In a failover scenario, you create multiple routing policies for the same DNS record. One is marked PRIMARY and others are marked SECONDARY. Route 53 continuously evaluates the health check associated with the primary. When the primary fails, traffic automatically shifts to the secondary.

Let me walk through a real example from a trading platform I built. We had:

• Primary: API running in us-east-1 (N. Virginia) with a health check on /api/health
• Secondary: API running in eu-west-1 (Ireland) with an identical health check
• DNS record: api.trading.example.com

When we queried api.trading.example.com, clients in normal conditions got the us-east-1 IP. If us-east-1's API started returning 500 errors, the health check failed. Within seconds, new DNS queries returned the eu-west-1 IP instead. Existing TCP connections needed to reconnect, but new requests went to the healthy region immediately.

Setting up failover routing via the AWS Management Console is straightforward, but here's conceptually what happens:

# Primary record (us-east-1)
Record Name: api.trading.example.com
Type: A
Value: 203.0.113.1
Routing Policy: Failover
Failover Record Type: Primary
Health Check: hc-east-1
Set ID: primary-east-1

# Secondary record (eu-west-1)
Record Name: api.trading.example.com
Type: A
Value: 198.51.100.1
Routing Policy: Failover
Failover Record Type: Secondary
Set ID: secondary-eu-west-1

When a failover occurs, the DNS name resolves to the secondary IP. Clients using your API must have retry logic or be prepared to refresh their DNS cache, which brings us to our next critical point.

Use Low TTLs for Faster Failover

TTL (Time To Live) determines how long DNS resolvers cache your answer. If your TTL is 300 seconds (5 minutes) and your primary fails, clients might not see the failover for up to 5 minutes. In mission-critical scenarios, that's unacceptable.

For high-availability setups, I always recommend a TTL between 30 and 60 seconds. This ensures that when a failover happens, most clients refresh their DNS cache and discover the secondary endpoint within a minute.

However, there's a trade-off: lower TTLs increase DNS query volume and costs. Route 53 charges per million queries, so dropping your TTL from 300 to 30 seconds increases your DNS query volume tenfold. In practice, I've found this cost acceptable for critical APIs (perhaps 30–40 extra queries per client per minute), but you should calculate it for your scale.

Here's an example of TTL impact:

My recommendation: use 60 seconds as a starting point. It balances reasonable failover speed with manageable DNS costs. For truly critical endpoints (payment processing, authentication), go as low as 30 seconds.

Test Failover Regularly

I've seen teams set up beautiful failover infrastructure that never actually worked when needed. The reason? They never tested it. Testing failover isn't optional; it's essential.

Here's how I approach failover testing:

1. Manual testing in non-production: In staging, intentionally stop the primary endpoint and verify that DNS queries resolve to the secondary. Use nslookup or dig to check DNS resolution.
2. Automated health check validation: Write a script that periodically checks that your health check endpoints are reachable and respond correctly. I've written Lambdas that do this and alert via SNS if something's wrong.
3. Chaos engineering in production: On a scheduled maintenance window, temporarily block traffic to your primary endpoint and observe failover behavior. Monitor your application metrics to ensure clients actually route to the secondary.
4. Failover drill documentation: Document what you expect to happen, measure what actually happens, and analyze any gaps. This teaches your team what failover looks like.

Here's a simple Lambda-based health check validator I've used:

import boto3
import requests

route53 = boto3.client('route53')
cloudwatch = boto3.client('cloudwatch')

def validate_health_check(health_check_id, endpoint_url):
    try:
        response = requests.get(endpoint_url, timeout=5)
        is_healthy = response.status_code == 200
    except Exception as e:
        is_healthy = False
        print(f"Health check failed: {e}")
    
    # Publish to CloudWatch
    cloudwatch.put_metric_data(
        Namespace='CustomHealthChecks',
        MetricData=[{
            'MetricName': 'EndpointHealthy',
            'Value': 1 if is_healthy else 0,
            'Dimensions': [{'Name': 'Endpoint', 'Value': endpoint_url}]
        }]
    )
    
    return is_healthy

Run this validator every minute via EventBridge, and you'll catch discrepancies between Route 53's view of health and reality.

Common Mistakes

Mistake 1: Not having a health check at all. Some teams create failover records but forget to attach health checks to the primary. Without a health check, Route 53 has no way to know the primary is down. Always explicitly configure health checks.

Mistake 2: Overly aggressive failure thresholds. Setting FailureThreshold to 1 means a single failed check triggers failover. On flaky networks, this causes unnecessary failovers. I typically use 3 as a minimum.

Mistake 3: Assuming clients retry or refresh DNS. If your primary fails but your client doesn't retry or clear its DNS cache, failover won't help. Build retry logic and DNS refresh into your client libraries. For mobile apps, I've used Squareup's OkHttp with aggressive DNS caching policies.

Mistake 4: Not monitoring failover events. If a failover happens but no one notices, you've lost visibility. Create CloudWatch alarms triggered by Route 53 health check status changes. Alert your team immediately.

Mistake 5: Ignoring asymmetric failover costs. If your primary region is much cheaper than your secondary (or vice versa), failover to an expensive region could spike costs. Plan for this and consider whether temporary cost increases are acceptable during outages.

Key Takeaways

• Health checks continuously monitor your endpoints. Route 53 uses their status to determine routing decisions.
• Failover routing automatically switches from primary to secondary when the primary health check fails.
• Lower TTLs (30–60 seconds) enable faster DNS-level failover but increase query volume and costs. Choose based on your criticality and budget.
• Test failover regularly in non-production and during controlled production drills. Untested failover is a liability, not an asset.
• Pair DNS failover with application-level retry logic. Clients should retry on connection failures or timeouts, not rely solely on DNS.
• Monitor health check status and failover events. Set up alerts so your team knows immediately when failures occur.