Circuit breaker: Design Patterns lessons from production

Circuit breaker: Design Patterns lessons from production

Your payment service calls an external gateway, which starts timing out. Within seconds, your entire checkout flow stalls as threads pile up waiting for responses that never come. By the time you notice, you've cascaded the failure across your system and your customers can't complete orders. The circuit breaker design pattern stops this before it happens by preventing your code from repeatedly calling a dependency that's already failing.

We've seen this problem firsthand in production across payment processors, shipping APIs, and third-party analytics platforms. A circuit breaker sits between your code and external services, monitoring success and failure rates. When failures exceed a threshold, it opens and stops sending traffic, returning fast failures instead of hanging requests. After a cool-down period, it cautiously tests the dependency with a few requests. If those succeed, it closes again and traffic flows normally. If they fail, it reopens and tries again later.

Understanding the three states

A circuit breaker operates in exactly three states, and understanding the transitions between them is fundamental to using the pattern effectively.

Closed: Normal operation

When the circuit is closed, calls flow through to the dependency as usual. The breaker tracks success and failure rates in the background. Every call increments either a success counter or failure counter. You define thresholds—typically something like "open the circuit if we see 5 failures in the last 10 requests" or "if the failure rate exceeds 50% over the last 100 calls."

In a production e-commerce system, your closed circuit might look like this: a request hits your inventory service, the circuit records a success, everything proceeds normally. This state should be your system's baseline; most of the time, your dependencies work fine and the circuit stays closed.

Open: Fast failure

Once failure thresholds are breached, the circuit opens. Now any attempt to call the dependency immediately returns an error without making the actual request. No threads block waiting for a doomed response. No connection pool slots get exhausted. The request fails quickly, and your code can handle it gracefully.

Consider a shipping API that's degraded. Without a circuit breaker, your checkout code might wait 30 seconds per call before timing out, and your thread pool fills up within seconds. With an open circuit, each call fails instantly, your code recovers quickly, and you can serve other customers or return a helpful message like "We're processing your order manually and will confirm delivery soon."

The open state protects your infrastructure from being exhausted by a failing upstream service. It's aggressive but necessary: you'd rather fail fast than waste resources.

Half-open: Testing recovery

An open circuit can't stay open forever. After a configurable timeout (typically 30–60 seconds), the circuit transitions to half-open. In this state, a small number of test requests are allowed through to the dependency. If those requests succeed, the circuit closes and normal operation resumes. If they fail, the circuit opens again and restarts the timeout timer.

This state is your system's way of asking: "Is the dependency working yet?" Without it, even if the external service recovered, your circuit would remain open forever and you'd never reconnect.

A practical example: your payment gateway had a database outage and recovered after two minutes. Your circuit was open for 60 seconds, then moved to half-open. The first three test requests succeeded, so the circuit closed. Customers can now pay again without any manual intervention.

Why circuit breakers matter in production

Retrying a dead service wastes resources in ways that compound rapidly. Consider a team without circuit breakers:

• Thread pools fill up with requests waiting for timeouts on a dead service
• Database connection pools get exhausted because threads hold them open
• Retry logic adds more requests to the queue, making things worse
• Memory usage spikes as queued requests accumulate
• Cascading failures spread to other services that depend on yours

In one incident we worked through, a partner's API went down for 8 minutes. The team's system kept retrying with exponential backoff, which sounds reasonable. But they had 200 concurrent users and a 30-second timeout. Within 60 seconds, 600+ threads were blocked waiting for that API. The application ran out of heap memory and crashed, taking down unrelated features. The cascade effect meant the entire platform went down, not just the feature using that API.

A circuit breaker would have opened after 30 seconds of failures, stopped wasting resources, and kept the rest of the platform running. Even though that specific feature was unavailable, the business could continue operating.

The pattern also provides visibility. When a circuit opens, that's a signal to alert your team, log the event, and investigate. You get early warning that something external is wrong, rather than discovering it when customers complain about slowness.

Implementing fallbacks alongside circuit breakers

A circuit breaker tells you when to stop calling a dependency, but it doesn't tell you what to return to the user. That's where fallbacks come in.

When your circuit opens or a half-open test fails, you have several options:

• Return cached data: If you know yesterday's product prices or inventory levels, return those rather than failing completely
• Provide a degraded experience: Show a simplified version of the feature without real-time data
• Return a friendly error: Tell the user "We're experiencing issues. Try again in a moment" rather than a technical 500 error
• Use a default value: For non-critical enrichment (like recommended products), return empty rather than failing the whole request

In a real scenario: your recommendation engine calls an ML service that has become unreliable. Instead of failing every product page, your circuit breaker catches the failure and your fallback returns an empty recommendations list. The page still loads, the user still completes their purchase, and you investigate the ML service during business hours rather than at 3 AM.

Fallbacks require planning. You need to know what data is safe to cache, which features are nice-to-have versus critical, and what users will tolerate. Discuss these scenarios with your product team before you need them.

Libraries and implementations

You don't need to build a circuit breaker from scratch. Mature libraries exist across platforms.

.NET: Polly

Polly is the standard circuit breaker library in .NET. It handles retries, timeouts, bulkheads, and circuit breaking in a fluent API.

var circuitPolicy = Policy
    .Handle<HttpRequestException>()
    .Or<TimeoutException>()
    .OrResult<HttpResponseMessage>(r => !r.IsSuccessStatusCode)
    .CircuitBreaker(
        handledEventsAllowedBeforeBreaking: 5,
        durationOfBreak: TimeSpan.FromSeconds(30),
        onBreak: (outcome, timespan) =>
        {
            Console.WriteLine($"Circuit opened for {timespan.TotalSeconds}s");
        },
        onReset: () =>
        {
            Console.WriteLine("Circuit reset, resuming calls");
        }
    );

var response = circuitPolicy.Execute(() => httpClient.GetAsync("https://api.example.com/data"));

This opens after 5 consecutive failures and stays open for 30 seconds. You can wrap multiple policies together and compose them with Policy.WrapAsync().

Node.js: Opossum

Opossum is Node's most popular circuit breaker. It's lightweight and integrates well with async/await.

import CircuitBreaker from "opossum";

const breaker = new CircuitBreaker(async () => {
  const response = await fetch("https://api.example.com/data");
  if (!response.ok) throw new Error("API error");
  return response.json();
}, {
  timeout: 3000,
  errorThresholdPercentage: 50,
  resetTimeout: 30000,
  rollingCountTimeout: 10000,
  rollingCountBuckets: 10,
  name: "payment-gateway",
  fallback: () => ({ cached: true, amount: 0 })
});

breaker.on("open", () => {
  console.log("Circuit opened, dependency failing");
});

breaker.on("halfOpen", () => {
  console.log("Testing recovery...");
});

try {
  const result = await breaker.fire();
} catch (err) {
  console.error("Request failed or circuit open:", err);
}

Notice the fallback option—that's your graceful degradation. When the circuit opens, opossum calls the fallback function instead of throwing.

Node.js alternative: Cockatiel

Cockatiel is a newer option with a composable API similar to Polly. It handles circuit breaking, retries, timeouts, and bulkheads.

import { CircuitBreaker, Backoff, HandleError } from "cockatiel";

const policy = CircuitBreaker.create({
  onOpen: () => console.log("Circuit open"),
  onHalfOpen: () => console.log("Circuit half-open"),
  onClose: () => console.log("Circuit closed"),
})
.pipe(
  new Backoff({ initialDelay: 100, maxDelay: 1000 })
)
.pipe(
  new HandleError({
    onFailure: (err) => console.error("Request failed:", err),
  })
);

const result = await policy.execute(() => fetchData());

Cockatiel's composable nature makes it easy to combine multiple resilience patterns without nesting callbacks.

Common mistakes

We've seen teams implement circuit breakers incorrectly in ways that reduce their effectiveness:

• Thresholds too high: Setting "open after 100 failures" means you've already damaged your system severely. Start conservative: 5–10 failures in a short window.
• Reset timeout too short: If you retry recovery after 5 seconds but the service won't be healthy for 2 minutes, you're just hammering it. Align timeout with expected recovery time.
• No monitoring on open events: A circuit opening is a critical signal. Log it, alert on it, don't let it be silent.
• Fallbacks that are worse than failure: Returning stale data that contradicts current state can confuse users more than a clear error message. Think through what degraded looks like.
• One circuit for multiple dependencies: Use separate circuits for separate external services. One bad API shouldn't trip the circuit for an unrelated service.
• Ignoring timeouts: A circuit breaker needs aggressive timeouts to work well. If you wait 60 seconds for a response before considering it a failure, the circuit will stay closed too long.

Key takeaways

• A circuit breaker transitions between Closed (normal), Open (fail fast), and Half-Open (test recovery) states based on failure rates
• Opening the circuit prevents cascading failures and resource exhaustion when dependencies are degraded
• Pair circuit breakers with fallbacks—decide upfront what your system does when the circuit opens
• Use battle-tested libraries: Polly for .NET, Opossum or Cockatiel for Node.js
• Alert when circuits open; these are production incidents that deserve visibility
• Configure thresholds conservatively: fail fast, don't wait for many failures to accumulate
• Align reset timeouts with realistic recovery times for your dependencies