Retries and timeouts with Resilience4j, explained simply

Introduction

Remote calls fail. Networks drop packets, services go down, databases hit their connection limits. A production system that doesn't handle these failures gracefully will cascade them into a full outage. On day 40 of our 100-day Java, Spring Framework & Spring Boot series, we're tackling retries and timeouts with Resilience4j, explained simply—because knowing when to retry and how long to wait is just as important as knowing that you should.

Resilience4j is a lightweight fault tolerance library built for the JVM. Unlike older tools like Hystrix, it uses Java 8 functional composition and works naturally with Spring Boot. We're going to walk through the patterns that matter: timeouts, retries, exponential backoff, and circuit breakers. If you get these right, your system stops falling apart when a dependency hiccups.

Timeouts first — retries without them make things worse

Before you think about retrying, set a timeout. A timeout is a maximum wall-clock duration that you're willing to wait for a response. Without one, a slow or hanging service can tie up all your threads, starving your entire application.

Consider a common scenario: your payment service calls an external processor. If that processor stops responding, your payment thread blocks indefinitely. Add a timeout, and your thread is freed after, say, 5 seconds. You can then fail fast, log the issue, and move on.

Here's how to set a timeout in Resilience4j with Spring Boot:

@Configuration
public class TimeLimitConfig {
    @Bean
    public TimeLimiter timeLimiter() {
        TimeLimiterConfig config = TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(5))
            .cancelRunningFuture(true)
            .build();
        return TimeLimiterRegistry.of(config).timeLimiter("payment-service");
    }
}

@Service
public class PaymentService {
    private final TimeLimiter timeLimiter;
    private final WebClient webClient;

    public PaymentService(TimeLimiter timeLimiter, WebClient webClient) {
        this.timeLimiter = timeLimiter;
        this.webClient = webClient;
    }

    public Mono<PaymentResponse> processPayment(PaymentRequest request) {
        return timeLimiter.executeMono(() -> 
            webClient.post()
                .uri("https://processor.external/charge")
                .bodyValue(request)
                .retrieve()
                .bodyToMono(PaymentResponse.class)
        );
    }
}

Without the timeout, a hanging HTTP call blocks your thread pool. With it, the request fails cleanly after 5 seconds. The cancelRunningFuture(true) flag ensures that if the underlying call supports cancellation, it's interrupted—not just abandoned.

Timeouts are often set too high or not set at all. A good rule of thumb: set your timeout to the 99th percentile of normal response time plus a small buffer. If your service typically responds in 200ms, a 5-second timeout is reasonable. If it's typically 100ms, go for 1–2 seconds.

Retry only idempotent operations

Not every failure deserves a retry. If you're charging a credit card and the call times out, you don't know if the charge went through. Retrying could charge the customer twice.

Idempotent operations are safe to retry because calling them multiple times has the same effect as calling them once. A GET request for user data is idempotent—asking for the same data 100 times returns the same result. A POST to create a payment is not idempotent by default unless you've added idempotency keys or designed the endpoint to deduplicate.

In Resilience4j, a Retry decorator will catch exceptions and re-invoke your operation. But only apply it to operations where you're certain a retry won't cause harm:

@Configuration
public class RetryConfig {
    @Bean
    public Retry retryPolicy() {
        RetryConfig config = RetryConfig.custom()
            .maxAttempts(3)
            .waitDuration(Duration.ofMillis(100))
            .retryExceptions(TimeoutException.class, SocketTimeoutException.class)
            .ignoreExceptions(PaymentAlreadyProcessedException.class)
            .build();
        return RetryRegistry.of(config).retry("idempotent-fetch");
    }
}

@Service
public class UserService {
    private final Retry retryPolicy;
    private final WebClient webClient;

    public UserService(Retry retryPolicy, WebClient webClient) {
        this.retryPolicy = retryPolicy;
        this.webClient = webClient;
    }

    public Mono<UserProfile> getUserProfile(String userId) {
        return Retry.executeSupplier(retryPolicy, () ->
            webClient.get()
                .uri("/users/{id}", userId)
                .retrieve()
                .bodyToMono(UserProfile.class)
        );
    }
}

Notice the retryExceptions and ignoreExceptions settings. We retry on transient errors like timeouts but not on application-level errors. If we get a 400 or 404, retrying won't help. Only transient, temporary failures are worth retrying.

Exponential backoff with jitter

If you retry immediately, you're likely to hit the same congestion that caused the failure in the first place. Worse, if many clients retry at the same time, you create a thundering herd—a spike in traffic that hammers a recovering service back down.

Exponential backoff means waiting longer between retries: 100ms, 200ms, 400ms, 800ms. Jitter adds randomness so that retries don't synchronize across all clients. Without jitter, every client retries at exactly the same intervals, recreating the thundering herd.

Resilience4j makes this straightforward:

@Configuration
public class BackoffConfig {
    @Bean
    public Retry retryWithBackoff() {
        RetryConfig config = RetryConfig.custom()
            .maxAttempts(5)
            .waitDuration(Duration.ofMillis(100))
            .intervalFunction(
                IntervalFunction.ofExponentialBackoff(
                    100,      // initial wait in ms
                    2,        // multiplier
                    0.1       // randomization factor (10% jitter)
                )
            )
            .retryExceptions(IOException.class, TimeoutException.class)
            .build();
        return RetryRegistry.of(config).retry("fetch-with-backoff");
    }
}

With this configuration:

• First failure: wait ~90–110ms (100ms ± 10%)
• Second failure: wait ~180–220ms
• Third failure: wait ~360–440ms
• And so on.

The jitter ensures that if 1,000 clients all fail at the same moment, they don't all retry at 100ms. Some retry at 95ms, some at 110ms, spreading the traffic spike.

In production, we've found that a multiplier of 2, an initial wait of 50–100ms, and jitter of 10% work well for most scenarios. Too aggressive a backoff delays your recovery; too conservative and you overwhelm the service.

Combine with a circuit breaker for failing dependencies

Retries help when a failure is temporary. But what if a service is down for an hour? Retrying for 5 attempts over 30 seconds is fine. Retrying for hours wastes resources and delays your own failures.

A circuit breaker sits in front of your retry logic. It tracks failure rates. If failures exceed a threshold, the circuit opens—calls fail immediately without attempting the remote call. After a timeout, it half-opens to test if the service has recovered.

Think of it like an electrical circuit breaker: when there's a surge, it trips and cuts power to protect the system. Once things stabilize, you can reset it.

Here's how to layer retries and circuit breakers in Resilience4j:

@Configuration
public class CircuitBreakerConfig {
    @Bean
    public CircuitBreaker circuitBreaker() {
        CircuitBreakerConfig config = CircuitBreakerConfig.custom()
            .failureRateThreshold(50)           // trip if 50% of calls fail
            .slowCallRateThreshold(50)          // trip if 50% are slow
            .slowCallDurationThreshold(Duration.ofSeconds(2))
            .waitDurationInOpenState(Duration.ofSeconds(30))
            .permittedNumberOfCallsInHalfOpenState(3)
            .recordExceptions(IOException.class, TimeoutException.class)
            .build();
        return CircuitBreakerRegistry.of(config).circuitBreaker("external-api");
    }

    @Bean
    public Retry retry() {
        RetryConfig config = RetryConfig.custom()
            .maxAttempts(3)
            .waitDuration(Duration.ofMillis(100))
            .intervalFunction(IntervalFunction.ofExponentialBackoff(100, 2, 0.1))
            .retryExceptions(IOException.class, TimeoutException.class)
            .build();
        return RetryRegistry.of(config).retry("external-api-retry");
    }

    @Bean
    public TimeLimiter timeLimiter() {
        TimeLimiterConfig config = TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofSeconds(5))
            .build();
        return TimeLimiterRegistry.of(config).timeLimiter("external-api");
    }
}

@Service
public class ExternalApiClient {
    private final CircuitBreaker circuitBreaker;
    private final Retry retry;
    private final TimeLimiter timeLimiter;
    private final WebClient webClient;

    public ExternalApiClient(CircuitBreaker circuitBreaker, Retry retry, 
                             TimeLimiter timeLimiter, WebClient webClient) {
        this.circuitBreaker = circuitBreaker;
        this.retry = retry;
        this.timeLimiter = timeLimiter;
        this.webClient = webClient;
    }

    public Mono<DataResponse> fetchData(String id) {
        return Mono.defer(() ->
            webClient.get()
                .uri("/data/{id}", id)
                .retrieve()
                .bodyToMono(DataResponse.class)
        )
        .transform(m -> timeLimiter.executeMono(() -> m))
        .transform(m -> retry.executeCompletableFuture(
            () -> m.toFuture()
        ).toMono().flatMap(r -> Mono.just(r)))
        .transform(m -> circuitBreaker.executeSupplier(() -> m.block()))
        .onErrorMap(ex -> new DataFetchException("Failed to fetch data", ex));
    }
}

The layering order matters: timeout → retry → circuit breaker. The timeout prevents slow calls from blocking forever. The retry handles transient failures. The circuit breaker stops you from hammering a dead service.

In practice, you'll see the circuit breaker state via metrics. Most monitoring tools can scrape Resilience4j metrics and alert you when a circuit opens.

Common mistakes

• No timeouts. A timeout is not optional. Without it, a slow service can exhaust your thread pool and cascade into a full outage.
• Retrying non-idempotent operations. Retrying a payment or money transfer without an idempotency key can charge the customer multiple times. Only retry operations you know are safe to retry.
• Retry without backoff. Immediate retries amplify thundering herd effects. Always use exponential backoff with jitter.
• Circuit breaker threshold too high. If your circuit breaker waits for 80% failure rate before opening, you'll be suffering for a long time. Set it lower, around 50%, so you fail fast on degraded dependencies.
• Ignoring retry metrics. If you see high retry counts in production, it's often a sign of upstream issues. Don't just ignore the metric; investigate.

Key takeaways

• Set a timeout before you retry. Timeouts prevent slow services from blocking your threads indefinitely.
• Only retry operations that are idempotent or have idempotency guards. Retrying arbitrary operations risks duplicating side effects like payments.
• Use exponential backoff with jitter. This prevents thundering herd effects and gives recovering services breathing room.
• Layer circuit breakers on top of retries. A circuit breaker stops you from wasting resources on a service that's clearly down.
• Monitor retry and circuit breaker state in production. High retry counts or frequent circuit openings signal upstream issues that need investigation.