Graceful shutdown in Spring Boot: lessons from production

Graceful shutdown in Spring Boot: lessons from production

We've all been there: a deployment happens, requests drop mid-flight, and suddenly you're fielding alerts about incomplete transactions. Graceful shutdown in Spring Boot is one of those features that seems optional until it costs you money. In this article, we'll walk through what we've learned deploying Spring Boot applications in production environments handling e-commerce orders, CRM sync jobs, and real-time trading updates.

Why graceful shutdown matters

When a Spring Boot application receives a termination signal (SIGTERM), it typically stops immediately. Active HTTP requests get cut off. Database transactions roll back. Message queue consumers disconnect mid-batch. For stateless read-heavy APIs, this might be acceptable. For anything processing money, state, or asynchronous work, it's a problem.

Graceful shutdown gives your application time to finish what it's doing before actually stopping. The Kubernetes orchestrator or reverse load balancer sends a termination signal. Your app stops accepting new requests, finishes the ones it has, and then shuts down cleanly. This is especially critical during rolling deployments where you're replacing instances one at a time.

Enable server.shutdown=graceful

The foundation is simple. In your application.yml or application.properties, add:

server:
  shutdown: graceful
  
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

This single property changes how Spring Boot behaves when it receives a shutdown signal. Instead of stopping immediately, it enters a graceful shutdown phase. But this is just the start.

Without this setting, Spring Boot stops synchronously. With it enabled, Spring Boot waits for all in-flight requests to complete before terminating the JVM. The magic happens in the embedded Tomcat (or your chosen servlet container) lifecycle management.

Set an appropriate shutdown timeout

Graceful shutdown isn't magic—it has limits. The timeout-per-shutdown-phase property controls how long Spring Boot will wait before forcing shutdown anyway. This needs to match your application's characteristics.

For a typical REST API with sub-second response times, 30 seconds is reasonable. For a payment processing service where transactions might take 10-15 seconds, consider 45 seconds. We've seen teams set this to 5 seconds and wonder why requests still drop—that's not enough time for even medium-complexity database queries.

Set the timeout based on your P99 request latency, not your average. Use your monitoring data:

• Look at your request latency distribution over the past month
• Find the 99th percentile
• Add 10-20% buffer for the occasional outlier
• Add another 5-10 seconds for the shutdown handoff itself

A concrete example: if your P99 latency is 8 seconds, set timeout-per-shutdown-phase: 20s. If you have long-running batch endpoints, increase it further or route them differently (more on that below).

You can also configure this per shutdown phase if your application has different requirements:

server:
  shutdown: graceful
  
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

Spring Boot has different shutdown phases (web container stop, bean destruction, etc.). Each phase gets the same timeout by default. This is usually fine, but be aware it applies uniformly.

Readiness probe must go down before shutdown

Here's a critical detail many teams miss: your readiness probe needs to start returning failure before actual shutdown begins. Otherwise, load balancers and orchestrators don't know the instance is going away.

When Kubernetes (or your orchestrator) sends a termination signal, here's the sequence we need:

1. Readiness probe immediately returns failure
2. Load balancer sees the instance is no longer ready and stops sending new requests
3. Graceful shutdown period begins; existing requests finish
4. After timeout or all requests complete, JVM exits

Without step 1, the load balancer might still send new requests even as the instance is shutting down. In Kubernetes, this happens because the readiness probe hasn't failed yet—the orchestrator waits for the probe to fail before removing the endpoint.

Implement a readiness probe that's aware of shutdown state:

@Component
public class ShutdownReadinessIndicator implements HealthIndicator {
    
    private final ApplicationContext context;
    private volatile boolean shuttingDown = false;
    
    public ShutdownReadinessIndicator(ApplicationContext context) {
        this.context = context;
        // Register for shutdown event
        context.publishEvent(new ApplicationStartedEvent(null, null, null, null));
    }
    
    @EventListener(ContextClosedEvent.class)
    public void onContextClosed() {
        shuttingDown = true;
    }
    
    @Override
    public Health health() {
        if (shuttingDown) {
            return Health.down()
                .withDetail("reason", "Application is shutting down")
                .build();
        }
        return Health.up().build();
    }
}

Then configure your readiness probe to hit the health endpoint:

management:
  endpoint:
    health:
      probes:
        enabled: true
  health:
    livenessState:
      enabled: true
    readinessState:
      enabled: true

In Kubernetes, your deployment YAML would look like:

readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 1

The failureThreshold: 1 means one failed probe immediately marks the instance as not ready. Combined with preStop hooks, this creates a clean handoff.

Gracefully handle background jobs and async tasks

HTTP requests aren't your only concern. If your application runs background jobs, scheduled tasks, or consumes from message queues, those need graceful handling too.

Consider a Spring Boot service consuming from Kafka. During shutdown, an in-flight message is being processed. Abrupt shutdown rolls back the transaction and the message goes back to the queue—duplicate processing on restart. Worse, if it's the 10th message of a batch, you've lost your processing position.

For scheduled tasks, implement proper cancellation:

@Component
public class ReportGenerationService {
    
    private final ExecutorService executor = Executors.newFixedThreadPool(4);
    private volatile boolean shutdown = false;
    
    @Scheduled(fixedDelay = 60000)
    public void generateDailyReport() {
        if (shutdown) {
            return;
        }
        // Process report
    }
    
    @PreDestroy
    public void shutdown() throws InterruptedException {
        shutdown = true;
        executor.shutdown();
        if (!executor.awaitTermination(25, TimeUnit.SECONDS)) {
            executor.shutdownNow();
            executor.awaitTermination(5, TimeUnit.SECONDS);
        }
    }
}

For Kafka consumers, Spring handles much of this automatically if you configure it properly:

spring:
  kafka:
    listener:
      ack-mode: manual_commit
      poll-timeout: 3000
    consumer:
      max-poll-records: 500
      
server:
  shutdown: graceful
  
spring:
  lifecycle:
    timeout-per-shutdown-phase: 30s

The manual commit strategy means messages are only marked as consumed after successful processing. During graceful shutdown, in-flight messages are processed before the consumer closes.

For long-running async jobs started by requests, you need to decide: finish or requeue. If a job takes 30 seconds and will be interrupted, can you save its state and resume? Implement a job service that tracks state:

@Component
public class AsyncJobService {
    
    private final JobRepository jobRepo;
    private final ExecutorService executor;
    private volatile boolean acceptingNewJobs = true;
    
    public String submitJob(JobRequest request) {
        if (!acceptingNewJobs) {
            throw new ServiceUnavailableException("Service is shutting down");
        }
        String jobId = UUID.randomUUID().toString();
        executor.submit(() -> processJob(jobId, request));
        return jobId;
    }
    
    @PreDestroy
    public void shutdown() throws InterruptedException {
        acceptingNewJobs = false;
        executor.shutdown();
        if (!executor.awaitTermination(25, TimeUnit.SECONDS)) {
            List unfinished = executor.shutdownNow();
            // Log unfinished work for monitoring
        }
    }
}

During graceful shutdown, new job submissions are rejected with a 503 error, signaling to the client to retry elsewhere. Existing jobs continue to completion.

Common mistakes

Setting shutdown timeout too low: Teams often use 5-10 seconds without measuring their actual request latencies. Measure P99, not average.

Not integrating readiness probes: Graceful shutdown only works if the load balancer stops sending traffic. Without readiness probe integration, new requests arrive during shutdown.

Forgetting about scheduled tasks: A scheduled batch job runs every 5 minutes, but shutdown timeout is 10 seconds. The job gets interrupted. Implement explicit shutdown handlers.

Not monitoring shutdown duration: Add metrics to track how long graceful shutdown actually takes. If it's consistently hitting the timeout, your timeout is too low or your requests are getting slower.

Ignoring preStop hooks: In Kubernetes, add a preStop hook that waits briefly before sending SIGTERM, giving the readiness probe time to propagate through the load balancer:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 10"]

Key takeaways

• Enable server.shutdown: graceful and set timeout-per-shutdown-phase based on your P99 request latency, not your average response time
• Integrate readiness probes so load balancers stop sending traffic before shutdown begins—this is essential in Kubernetes environments
• Explicitly handle background jobs, scheduled tasks, and message consumers with @PreDestroy methods that wait for existing work to complete
• Monitor the actual duration of graceful shutdowns in production; if it consistently hits the timeout, investigate whether requests are slowing down
• Test shutdown behavior in staging by simulating rolling deployments and verifying that in-flight requests complete cleanly