High availability patterns: AWS best practices explained

High availability patterns: AWS best practices explained

We've all been on-call when a single server crash took down the entire platform for thirty minutes. It shouldn't happen. High availability patterns are the architectural decisions that let you sleep at night, knowing that one component failing doesn't cascade into an outage. In this article, we'll walk through the core patterns we use to design systems where failures are isolated, requests reroute automatically, and your service stays up even when pieces break.

Why availability matters more than you think

Every minute your service is down costs money, erodes user trust, and generates pages to your on-call engineer at 3 a.m. High availability isn't about perfection—it's about designing systems that tolerate failure as a normal, expected event. AWS gives us the tools to do this at every layer: compute, storage, networking, and beyond.

The patterns we'll cover today form the foundation of production systems across e-commerce, SaaS, and real-time platforms. They're not optional complexity; they're the baseline expectation.

Multiple AZs—Spread compute and databases across zones

AWS regions are subdivided into Availability Zones (AZs), each with independent power, cooling, and networking. A single AZ can go down—and they do. We've seen network storms, power incidents, and maintenance windows take out entire zones. The solution is straightforward: never put all your eggs in one basket.

For stateless services like API servers or workers, this means running at least two instances across different AZs behind a load balancer. When one AZ goes down, traffic automatically shifts to the other. For RDS databases, the same principle applies: use Multi-AZ deployments where the primary and standby replicas sit in separate zones. Failover happens automatically.

Here's what a typical setup looks like in Terraform:

resource "aws_autoscaling_group" "api_servers" {
  name            = "api-asg"
  min_size        = 2
  max_size        = 6
  desired_capacity = 2
  
  vpc_zone_identifier = [
    aws_subnet.private_az1.id,
    aws_subnet.private_az2.id,
    aws_subnet.private_az3.id
  ]
  
  launch_template {
    id      = aws_launch_template.api.id
    version = "$Latest"
  }
}

resource "aws_db_instance" "postgres" {
  allocated_storage    = 100
  engine              = "postgres"
  multi_az            = true
  availability_zone   = aws_subnet.private_az1.availability_zone
  
  backup_retention_period = 30
}

Notice the Auto Scaling Group spans three AZs. If one zone has a problem, the group can still serve from the other two. The RDS instance has multi_az = true, which means AWS maintains a synchronous replica in a different zone.

The cost overhead is modest—you're buying redundancy, not capacity. If your service needs two instances anyway, spreading them across AZs is the first thing you do.

Stateless services—Any instance can serve any request

Statefulness kills availability. When a user's session lives only on server A, and server A crashes, that user can't connect. We've seen this pattern destroy otherwise solid architectures.

Stateless design means every instance of your service is identical and interchangeable. A user can hit instance 1 on their first request and instance 3 on their second. To achieve this, you externalize state: session data, temporary request tokens, feature flags, anything that varies per user or request goes into a shared store.

Here's a typical stateless API pattern:

// Node.js / NestJS example
import { Controller, Get, Req } from '@nestjs/common';
import { RedisService } from './redis.service';

@Controller('/api/user')
export class UserController {
  constructor(private redis: RedisService) {}

  @Get('/profile')
  async getProfile(@Req() req) {
    // Read session from Redis, not memory
    const sessionId = req.cookies.sessionId;
    const userId = await this.redis.get(`session:${sessionId}`);
    
    if (!userId) {
      throw new UnauthorizedException();
    }
    
    // Fetch user data from database
    const user = await this.userService.findById(userId);
    return user;
  }
}

Every instance queries the same Redis cluster for session data. When you kill an instance, in-flight requests fail gracefully (which the load balancer catches), and the next request goes to a healthy instance. No sticky sessions, no data loss, no cascading failures.

Caching layers like ElastiCache (Redis or Memcached) become critical infrastructure. Size them properly and monitor hit rates. A cache miss storm can melt your database.

Health checks everywhere—Load balancers route around failures

A crashed instance that still responds to TCP pings will keep receiving traffic until the load balancer detects the problem. Health checks are how we automate failure detection and rerouting.

Application Load Balancers (ALBs) in AWS support HTTP health checks: they periodically hit an endpoint on each target instance and expect a specific response. If an instance fails the check repeatedly, the ALB stops sending traffic to it. If the whole instance fails, that's caught too.

Set up a dedicated health check endpoint that's fast and deterministic:

// Express.js example
app.get('/_health', async (req, res) => {
  try {
    // Check database connectivity
    await db.query('SELECT 1');
    
    // Check cache connectivity
    await redis.ping();
    
    res.status(200).json({ status: 'healthy' });
  } catch (error) {
    // Return 503 if dependencies are unhealthy
    res.status(503).json({ status: 'unhealthy', error: error.message });
  }
});

The health endpoint checks dependencies: database, cache, external APIs if they're critical. If any dependency is unreachable, return HTTP 503 (Service Unavailable). The ALB sees the 503 and marks the instance unhealthy.

Configure the ALB like this:

resource "aws_lb_target_group" "api" {
  name            = "api-tg"
  port            = 3000
  protocol        = "HTTP"
  vpc_id          = aws_vpc.main.id
  
  health_check {
    healthy_threshold   = 2
    unhealthy_threshold = 3
    timeout             = 5
    interval            = 30
    path                = "/_health"
    matcher             = "200"
  }
  
  deregistration_delay = 30
}

The thresholds matter: an instance needs 3 failed checks before the ALB removes it. This prevents flapping (rapid add/remove) from brief network hiccups. The 30-second deregistration delay gives in-flight requests time to complete before forcefully closing connections.

Health checks should run fast—under 5 seconds—because slow checks delay failure detection. And always check your actual dependencies, not just "is the process running?"

Graceful degradation—Serve cached data when dependencies fail

Sometimes a database connection dies or an external API times out. Rather than returning a 500 error to the user, we serve stale-but-useful data from cache. This keeps the service up even when its dependencies are partially broken.

The pattern is simple: cache success responses aggressively, and on failure, return the cached version if it exists.

// Example: fetching product details
async function getProductDetails(productId: string) {
  const cacheKey = `product:${productId}`;
  
  try {
    // Try the database first
    const product = await db.query(
      'SELECT * FROM products WHERE id = $1',
      [productId]
    );
    
    // Cache the result for 5 minutes
    await redis.setex(cacheKey, 300, JSON.stringify(product));
    return product;
  } catch (error) {
    // Database failed; return cached data if available
    const cached = await redis.get(cacheKey);
    
    if (cached) {
      // Return stale data but mark it as such
      return {
        ...JSON.parse(cached),
        _stale: true,
        _error: 'Using cached data; database unavailable'
      };
    }
    
    // No cache and database failed; now we error
    throw error;
  }
}

The client gets data either way. On the UI, you might show a subtle indicator ("This data may be outdated") but the user can still browse. For product pages in e-commerce, this is often acceptable—showing an old price is better than a 500 error.

Graceful degradation isn't about hiding failures. It's about choosing what you want to fail: would you rather lose the ability to show new ratings, or lose the entire product page? Cache ratings aggressively and degrade that feature if the database is slow.

Common mistakes

Mistake 1: Multi-AZ database without multi-AZ compute. You set up RDS Multi-AZ to be safe, but all your application servers run in AZ1. When AZ1 fails, your database still works, but your apps don't. Always spread compute and data across the same zones.

Mistake 2: Health checks that don't check dependencies. A health endpoint that returns 200 just because the process is running is useless. If the database is down, the service is down. Make the health check reflect reality.

Mistake 3: Overconfidence in cache. Caching everything forever seems safe until you push a database schema change and need to flush the cache. Set sensible TTLs and invalidate explicitly when you know data has changed.

Mistake 4: Forgetting about data consistency in graceful degradation. If you serve cached product prices, you'd better document that to users. Selling something at a six-month-old price is a problem. Graceful degradation works best for read-heavy, slowly-changing data.

Mistake 5: Single points of failure in "redundant" setups. Two load balancers across AZs are good, but if they both use the same Route 53 record and Route 53 has a query limit you hit, the whole thing fails. Think beyond the obvious layer.

Key takeaways

• Spread compute and databases across at least two AZs. Assume one zone will fail without warning.
• Design services to be stateless. Use external stores (Redis, RDS) for anything a user or request needs to persist between calls.
• Implement health checks on all load balancers. Check your actual dependencies, not just whether the process is alive.
• Cache aggressively and return stale data on dependency failure. Degraded availability beats unavailability.
• Test failover. Kill instances, block database connections, and watch your system recover. Untested high availability is fantasy.
• Monitor with actual metrics, not just uptime pings. Track error rates, latency percentiles, and cache hit rates to catch problems before users do.

High availability is a mindset as much as a checklist. Every architectural decision should ask: what happens when this fails? The answer shouldn't be "the whole system crashes."