Retry with exponential backoff: best practices explained

Introduction

Network calls fail. Databases go down for maintenance. Rate limits kick in. When building distributed systems, transient failures are inevitable, and blindly retrying every error will only make things worse. Retry with exponential backoff: best practices explained covers the pattern that separates production-ready systems from those that amplify failures into cascading outages.

We've all seen the aftermath of poorly implemented retry logic: thundering herds of requests overwhelming a struggling service, infinite loops consuming resources, or retrying operations that should never be retried. Getting this right requires understanding not just the mechanics of backoff, but the constraints that make retries safe and effective.

Retry only transient failures

The first and most critical decision is knowing when to retry. Not every error deserves a second attempt. Some failures are permanent—a 404 Not Found, a 403 Forbidden, a validation error—and retrying these wastes time and resources. Transient failures, on the other hand, are temporary: a 503 Service Unavailable, a network timeout, a brief connection reset.

The distinction matters because retrying a permanent failure doesn't change the outcome. If a database query is malformed, running it again won't fix it. But if a service is momentarily overwhelmed, waiting a moment and retrying often succeeds.

In practice, classify failures by HTTP status code or exception type. Consider this categorization:

• Transient (retry): 408 Request Timeout, 429 Too Many Requests, 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout, network errors like ECONNRESET or ETIMEDOUT
• Permanent (don't retry): 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, validation errors, parsing errors

Create a helper to identify retryable errors:

function isTransient(error: any): boolean {
  if (error.statusCode) {
    return [408, 429, 500, 502, 503, 504].includes(
      error.statusCode
    );
  }
  if (error.code) {
    return ['ECONNRESET', 'ETIMEDOUT', 'ENOTFOUND'].includes(
      error.code
    );
  }
  return false;
}

async function safeRetry<T>(
  fn: () => Promise<T>,
  tries = 4
): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (e) {
      if (!isTransient(e) || i >= tries - 1) {
        throw e;
      }
      await sleep(calculateBackoff(i));
    }
  }
}

By failing fast on permanent errors, you avoid wasted retry attempts and surface problems to callers immediately. This is especially important in request-response cycles where the caller is waiting.

Exponential backoff with jitter

Blindly retrying immediately after a failure is counterproductive. If a service is struggling under load, hammering it with instant retries accelerates its collapse. Exponential backoff introduces progressively longer delays between attempts, giving the failing service time to recover.

The formula is simple: delay = base × (2 ^ attempt). For the second attempt, wait 2 units. For the third, wait 4 units. For the fourth, wait 8 units. This allows the system breathing room without waiting excessively long.

However, exponential backoff introduces a new problem: the thundering herd. If thousands of clients all experience a transient failure at the same moment, they'll all retry after the same delay, creating a synchronized spike that re-triggers the failure. The solution is jitter—adding randomness to the delay.

Instead of waiting exactly 2 ^ attempt seconds, add a random value. Common approaches include full jitter and equal jitter. We'll use a straightforward version: multiply the exponential value by a random factor between 0.5 and 1.5, or add a random offset.

function calculateBackoff(
  attempt: number,
  baseMs: number = 200,
  maxMs: number = 30000
): number {
  // Exponential: 200ms, 400ms, 800ms, 1600ms...
  const exponential = baseMs * Math.pow(2, attempt);
  
  // Cap maximum delay
  const capped = Math.min(exponential, maxMs);
  
  // Add jitter: random value between 0 and capped
  const jitter = Math.random() * capped;
  
  return capped + jitter;
}

async function retryWithBackoff<T>(
  fn: () => Promise<T>,
  tries = 4
): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (e) {
      if (!isTransient(e) || i >= tries - 1) {
        throw e;
      }
      const delayMs = calculateBackoff(i);
      await sleep(delayMs);
    }
  }
}

Jitter prevents synchronized retries across distributed clients. It smooths out the load spike and significantly improves the chances that retries will succeed. Without it, coordinated failures become worse, not better.

Cap the number of attempts

Retries must have an upper limit. Without a cap, a permanently failing operation can retry indefinitely, consuming resources and time. Your retry logic needs a circuit breaker: after N attempts, give up and propagate the error.

How many attempts is reasonable? It depends on your use case. For user-facing requests, 3–4 attempts is typical (base delay of 200ms means total wait time of ~1–2 seconds). For background jobs, you might retry more aggressively since the caller isn't blocking. For external APIs, fewer retries might be appropriate since they're outside your control.

The code above already implements this with the tries parameter. On the final attempt, any error is thrown immediately without waiting. This gives callers a predictable failure time and prevents resource exhaustion.

// User-facing request: 4 attempts, ~2 second max wait
const result = await retryWithBackoff(
  () => fetchUserData(userId),
  4
);

// Background job: 8 attempts, more tolerance
await retryWithBackoff(
  () => processPayment(orderId),
  8
);

When setting attempts, consider your SLA and the impact of failure. If a user can tolerate a 5-second delay, that might allow 4–5 attempts. If you're talking to an external API with strict rate limits, fewer attempts protect you from triggering abuse detection.

Only retry idempotent operations

This is the constraint that makes or breaks retry safety. An idempotent operation produces the same result no matter how many times it's executed. Reading data is idempotent. Writing the same record twice should be safe only if you design it that way.

Retrying a non-idempotent operation is dangerous. Imagine debiting an account, the network goes down, and the client retries. If the initial debit succeeded before the connection failed, retrying creates a duplicate charge. This is catastrophic and invisible until the customer notices.

Design your API endpoints for idempotency by using unique request identifiers. When a request arrives, check if you've already processed it; if so, return the cached result instead of processing again:

// Client generates or provides idempotency key
const idempotencyKey = crypto.randomUUID();

async function retryableDebit(
  accountId: string,
  amount: number,
  idempotencyKey: string
): Promise<TransactionResult> {
  // Check if we've already processed this
  const existing = await db.query(
    'SELECT * FROM transactions WHERE idempotency_key = $1',
    [idempotencyKey]
  );
  if (existing) {
    return existing;
  }

  // Process for the first time
  const result = await db.transaction(async (tx) => {
    await tx.query(
      'UPDATE accounts SET balance = balance - $1 WHERE id = $2',
      [amount, accountId]
    );
    return tx.query(
      'INSERT INTO transactions (account_id, amount, idempotency_key) VALUES ($1, $2, $3)',
      [accountId, amount, idempotencyKey]
    );
  });

  return result;
}

// Now safe to retry
const debit = await retryWithBackoff(
  () => retryableDebit(accountId, 100, idempotencyKey),
  4
);

For read-only operations (fetching data), you're already safe. For operations that mutate state, either design them as idempotent (using unique keys) or accept the risk of duplicates. Never silently retry mutations that aren't idempotent.

Common mistakes

Retrying everything: Retrying a 400 Bad Request wastes time. Fail fast on permanent errors. Use isTransient() checks religiously.

Exponential backoff without jitter: Creates synchronized retry storms. Always add randomness to your delays, especially in distributed systems.

No cap on attempts: A retry loop with no upper bound can hang indefinitely. Always specify maxAttempts and throw after exhaustion.

Retrying non-idempotent operations: Double-charges, duplicate inserts, and data corruption happen here. Only retry operations designed for idempotency, or accept the risk explicitly.

Ignoring the context: A 4-attempt retry is fine for internal services but might be too aggressive for external APIs. Tune your retry strategy to your environment.

Key takeaways

• Classify errors as transient or permanent; only retry transient failures like 503 or timeouts
• Use exponential backoff to give failing services recovery time, and add jitter to prevent thundering herds
• Always cap the number of retry attempts; 3–4 is typical for user-facing requests
• Only retry operations that are idempotent or explicitly handle duplicate executions to avoid data corruption
• Monitor retry rates and success rates in production to tune your strategy