Failure is a feature: designing resilient systems explained

Failure is a Feature: Designing Resilient Systems

In five years of building production platforms—from e-commerce to ride-hailing—I've learned one unchangeable truth: every external dependency will fail, and often at the worst possible moment. The difference between a system that stumbles and one that gracefully absorbs failure comes down to how deliberately you plan for it. Resilience isn't an afterthought or a premium feature; it's a foundational design decision that separates mature platforms from fragile ones.

When I say "failure is a feature," I mean this literally. You don't add resilience after outages happen; you architect for failure the same way you architect for correctness. A resilient system doesn't hide failures—it detects them early, handles them gracefully, and keeps serving value to users even when parts break.

Timeouts Everywhere — No Outbound Call Without a Deadline

The first rule I enforce on every team: there is no outbound network call without a timeout. None. This includes HTTP requests, database queries, message queue operations, and cache lookups.

Why? Because without a timeout, a hanging dependency can consume threads indefinitely, eventually starving your entire application. I've watched a single slow third-party API bring down an e-commerce checkout service because timeout wasn't configured. Requests piled up, connection pools exhausted, and new requests failed immediately.

The timeout value itself matters. Set it too high and you're just delaying the pain. Set it too low and you'll flake out on legitimate requests under load. I typically recommend:

• External APIs: 5-10 seconds, depending on SLA
• Internal microservices: 2-5 seconds in same datacenter, longer for cross-region
• Database queries: 30 seconds for normal queries, 5 seconds for critical path operations
• Cache operations: 500ms (fail fast and move to source)

Here's a practical pattern using NestJS with Axios:

import axios from 'axios';

const httpClient = axios.create({
  timeout: 5000, // 5 second timeout
  // Important: timeout applies to entire request, not just connection
});

async function fetchUserData(userId: string) {
  try {
    const response = await httpClient.get(
      `https://external-api.com/users/${userId}`,
      { timeout: 5000 }
    );
    return response.data;
  } catch (error) {
    if (error.code === 'ECONNABORTED') {
      // Timeout occurred
      console.error('User API timeout after 5s');
      throw new TimeoutError('External service unresponsive');
    }
    throw error;
  }
}

Notice I set timeout in two places: the client config and the individual request. This defense-in-depth approach ensures no call can hang, even if a config gets missed later.

Retries With Backoff — Retry Transient Failures Only, With Jitter and Limits

Not every failure is permanent. Network hiccups, temporary service degradation, and brief outages are transient. The trick is retrying only those failures, with exponential backoff and jitter to avoid thundering herd problems.

The critical decision: what failures are retryable? Only idempotent operations on transient errors. Never retry write operations unless you're certain the operation is idempotent (like upserts). Never retry permanent errors like 400 Bad Request or 403 Forbidden. Safe retries are usually:

• GET requests (always safe to retry)
• Transient HTTP status codes: 408 (timeout), 429 (rate limit), 502/503/504 (gateway/service unavailable)
• Network errors: connection refused, DNS failures (but not after timeout)

Here's a robust retry implementation with exponential backoff and jitter:

async function retryWithBackoff(
  operation: () => Promise,
  maxRetries: number = 3,
  baseDelayMs: number = 100
): Promise {
  let lastError: Error;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      lastError = error;
      const isTransient =
        error.code === 'ECONNREFUSED' ||
        error.code === 'ECONNABORTED' ||
        error.code === 'ETIMEDOUT' ||
        (error.response?.status >= 500 && error.response?.status < 600) ||
        error.response?.status === 429;

      const isLastAttempt = attempt === maxRetries;

      if (!isTransient || isLastAttempt) {
        throw error;
      }

      // Exponential backoff: 100ms, 200ms, 400ms
      const exponentialDelay = baseDelayMs * Math.pow(2, attempt);
      // Jitter: add randomness to prevent thundering herd
      const jitter = Math.random() * exponentialDelay;
      const delayMs = exponentialDelay + jitter;

      console.log(
        `Retry attempt ${attempt + 1}/${maxRetries} after ${delayMs}ms`
      );
      await new Promise(resolve => setTimeout(resolve, delayMs));
    }
  }

  throw lastError;
}

// Usage
const userData = await retryWithBackoff(
  () => fetchUserData(userId),
  3,
  100
);

The jitter is important: without it, if 1000 services all retry after 1 second, they hammer the recovering service simultaneously. Jitter spreads them out naturally.

Circuit Breakers — Stop Hammering Dependencies That Are Already Down

Retries help with transient failures, but they make cascading failures worse. If a dependency is genuinely down, retrying just wastes resources and prolongs the outage for your users. Enter the circuit breaker pattern.

A circuit breaker has three states:

• Closed (normal): Requests flow through. Track failures.
• Open (failure detected): Stop sending requests. Fail fast immediately.
• Half-open (recovery test): Allow a single request through to test if the dependency recovered.

I used this pattern heavily in our ride-hailing platform when payment processing went down. Without a circuit breaker, every ride completion would retry payment calls dozens of times, overwhelming the payment service and delaying recovery. With it, we failed fast after the first few attempts and served a cached experience.

Here's a practical implementation using the popular opossum library:

import CircuitBreaker from 'opossum';

const options = {
  timeout: 5000,
  errorThresholdPercentage: 50, // Trip if 50% of requests fail
  resetTimeout: 30000, // Try recovery after 30s
  name: 'payment-service',
  rollingCountTimeout: 10000, // Track stats over 10s window
};

const paymentBreaker = new CircuitBreaker(
  async (amount: number, userId: string) => {
    return await httpClient.post('/charge', { amount, userId });
  },
  options
);

// Fallback when circuit is open
paymentBreaker.fallback(() => ({
  status: 'pending',
  message: 'Payment service temporarily unavailable. Will retry.',
}));

// Usage
try {
  const result = await paymentBreaker.fire(amount, userId);
  console.log('Payment processed:', result);
} catch (error) {
  if (error.message.includes('breaker is open')) {
    console.log('Circuit breaker is open, returning fallback');
  } else {
    throw error;
  }
}

Monitor the circuit breaker state in production. When it opens, alert your team—that's a signal that a dependency is struggling and needs investigation.

Fallbacks — Serve Cached Data or a Reduced Experience Instead of an Error Page

The final layer: when a dependency fails despite timeouts, retries, and circuit breakers, what do you return to the user? An error page is the wrong answer almost always.

Instead, design fallback responses that degrade gracefully. In an e-commerce site, if the recommendation service is down, show popular products. In a travel booking app, if real-time pricing is slow, serve cached prices from 15 minutes ago with a "prices may be outdated" badge. Users will tolerate degradation; they won't tolerate broken pages.

Here's a realistic example from a CRM platform I built:

async function getUserDashboard(userId: string) {
  const cacheKey = `dashboard:${userId}`;

  try {
    // Try to fetch fresh data with timeout
    const freshData = await retryWithBackoff(
      () => fetchDashboardData(userId),
      2,
      50
    );
    // Cache for 5 minutes
    await redis.setex(cacheKey, 300, JSON.stringify(freshData));
    return { data: freshData, cached: false, stale: false };
  } catch (error) {
    console.error('Failed to fetch fresh dashboard data:', error);

    // Fallback 1: Serve cached data
    const cachedData = await redis.get(cacheKey);
    if (cachedData) {
      return {
        data: JSON.parse(cachedData),
        cached: true,
        stale: true, // Signal to UI that data may be old
      };
    }

    // Fallback 2: Serve minimal skeleton
    return {
      data: getSkeletonDashboard(userId),
      cached: false,
      stale: true,
      message: 'Showing minimal dashboard. Service will be back soon.',
    };
  }
}

Notice the stale flag. The UI can show a subtle banner: "Showing data from 5 minutes ago" instead of silently serving stale data. Transparency builds trust.

Common Mistakes

• Setting timeouts too high: A 30-second timeout on an API call doesn't protect you; it just delays the failure and burns resources. Better to fail fast after 5-10 seconds than to watch threads pile up.
• Retrying non-idempotent operations: Retrying a payment charge without idempotency keys is how you charge users twice. Always think through the retry semantics.
• Circuit breaker thresholds that are too aggressive: If you trip the breaker after just 3 failures, legitimate slow requests will trigger it constantly. Use percentages, not absolute counts, and give the breaker at least 10-100 samples before deciding.
• No observability on resilience mechanisms: If you don't log timeouts, retries, and circuit breaker state changes, you're flying blind. Instrument everything.
• Forgetting about resource cleanup: Retries and timeouts can leave hanging connections or pending timers. Always ensure your HTTP clients pool connections properly and clean up on timeout.

Key Takeaways

• Assume every external dependency will fail. Design your system to detect and handle failure gracefully, not as an edge case.
• Every outbound call needs a timeout. Without it, hanging dependencies will starve your application's resources.
• Retry transient failures only, using exponential backoff with jitter to avoid thundering herd problems on recovery.
• Use circuit breakers to fail fast when a dependency is genuinely down, preventing cascading failures across your platform.
• Design fallbacks into every critical user path: cached data, reduced features, or skeletons are better than error pages.
• Instrument your resilience mechanisms so you can monitor and alert on timeouts, retries, and circuit breaker trips.
• Resilience is not optional. It's the difference between a platform your users trust and one they avoid.