Graceful shutdown in Go: real-world lessons and examples

Why graceful shutdown matters in production

You've deployed your Go API, traffic is flowing, and then it's time to roll out a new version or scale down a service. If you just kill the process, you'll lose in-flight requests, corrupt data writes, and disappoint users. Graceful shutdown in Go: real-world lessons and examples shows us how to give running requests time to finish before the process exits. This matters because databases, message queues, and downstream services expect clean disconnections, and your team will thank you when deploys don't cascade into partial failures.

Listen for signals with signal.NotifyContext

The foundation of graceful shutdown is catching the termination signals your OS sends. When you run docker stop, send SIGTERM, or press Ctrl+C during local development, your process receives a signal. Go's signal package lets us intercept these and react deliberately instead of crashing.

The cleanest modern approach is signal.NotifyContext, introduced in Go 1.16. It wraps a context and cancels it when a signal arrives. Here's the pattern:

package main

import (
	"context"
	"net/http"
	"os"
	"os/signal"
	"syscall"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	srv := &http.Server{Addr: ":8080"}

	// Start server in a goroutine
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			panic(err)
		}
	}()

	// Wait for signal
	<-ctx.Done()
	println("signal received, shutting down")

	// Now proceed to graceful shutdown
	srv.Shutdown(context.Background())
}

In this example, we listen for os.Interrupt (Ctrl+C) and syscall.SIGTERM (the standard termination signal). The defer stop() is important—it stops listening for signals once we've handled one, preventing duplicate shutdown logic.

This approach is cleaner than the older signal.Notify pattern because the context integrates naturally with the rest of your application logic. You can pass the context downstream to workers, health checks, and database operations.

Shutdown the server without losing requests

Once we've caught the signal, we need to tell the HTTP server to stop accepting new connections but finish in-flight requests. The http.Server.Shutdown method does exactly this.

Here's what happens when you call Shutdown:

1. The listener stops accepting new connections.
2. Open connections are kept alive temporarily.
3. In-flight requests are allowed to complete.
4. The method blocks until all requests finish or a timeout occurs.

Let's build a more realistic example with a router and some endpoints:

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	// Set up signal handling
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	// Create router
	mux := http.NewServeMux()
	mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "ok")
	})
	mux.HandleFunc("/process", func(w http.ResponseWriter, r *http.Request) {
		// Simulate work
		time.Sleep(2 * time.Second)
		w.WriteHeader(http.StatusOK)
		fmt.Fprintf(w, "processed")
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	// Start server in goroutine
	go func() {
		log.Printf("starting server on %s", srv.Addr)
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Fatalf("listen error: %v", err)
		}
	}()

	// Block until signal
	<-ctx.Done()
	log.Println("signal received, starting graceful shutdown")

	// Shutdown with a timeout
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()

	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Fatalf("shutdown error: %v", err)
	}

	log.Println("shutdown complete")
}

Notice we're already introducing the concept of a shutdown timeout (covered in the next section). This prevents Shutdown from hanging forever if a request gets stuck.

Set a timeout for graceful shutdown

In an ideal world, all in-flight requests finish quickly. In reality, some slow queries, hung connections, or buggy handlers can block shutdown indefinitely. A timeout ensures your deployment doesn't hang for hours waiting for a misbehaving goroutine.

We pass a context with a deadline to Shutdown:

shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

if err := srv.Shutdown(shutdownCtx); err != nil {
	if err == context.DeadlineExceeded {
		log.Println("shutdown timeout exceeded, some requests may be killed")
	}
	log.Fatalf("shutdown error: %v", err)
}

Thirty seconds is a reasonable default for most services, but you might adjust based on your workload. E-commerce systems processing payments might use 60 seconds; real-time APIs might use 10.

If the timeout expires before all requests finish, Shutdown returns a context.DeadlineExceeded error, and any remaining connections are forcibly closed. This is not ideal, but it's better than a deploy that never finishes.

A practical tip from our production experience: log what happens at shutdown. Did you hit the timeout? How many requests were still running? This data helps you tune the timeout and spot problematic endpoints.

shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

start := time.Now()
if err := srv.Shutdown(shutdownCtx); err != nil {
	log.Printf("shutdown took %v, error: %v", time.Since(start), err)
}
log.Printf("graceful shutdown completed in %v", time.Since(start))

Stop background workers and goroutines

Most real APIs don't just serve HTTP—they run background jobs, consume from message queues, or maintain database connections. These need graceful shutdown too.

The pattern is straightforward: pass the same signal context to your workers, and have them check for cancellation:

package main

import (
	"context"
	"log"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	srv := &http.Server{Addr: ":8080"}
	var wg sync.WaitGroup

	// Start HTTP server
	go func() {
		if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
			log.Printf("server error: %v", err)
		}
	}()

	// Start background worker
	wg.Add(1)
	go func() {
		defer wg.Done()
		backgroundWorker(ctx)
	}()

	// Wait for signal
	<-ctx.Done()
	log.Println("shutdown signal received")

	// Shutdown HTTP server
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
	defer cancel()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("server shutdown error: %v", err)
	}

	// Wait for workers to finish
	done := make(chan struct{})
	go func() {
		wg.Wait()
		close(done)
	}()

	select {
	case <-done:
		log.Println("all workers stopped gracefully")
	case <-time.After(10 * time.Second):
		log.Println("worker shutdown timeout exceeded")
	}
}

func backgroundWorker(ctx context.Context) {
	ticker := time.NewTicker(5 * time.Second)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			log.Println("worker shutting down")
			return
		case <-ticker.C:
			// Do some background work
			log.Println("worker processing")
		}
	}
}

Key points in this pattern:

• Use sync.WaitGroup to track all background goroutines.
• Pass the signal context to each worker.
• Workers check <-ctx.Done() in their main loop.
• After signaling workers to stop, wait for them with a timeout of their own.

This ensures that when your container gets the stop signal, every part of your application gets a chance to clean up: flush buffers, commit transactions, close connections, drain queues.

Common mistakes

Ignoring the shutdown error. A common pattern is defer srv.Shutdown(context.Background()). This swallows errors and uses an infinite timeout. Better to explicitly handle it and log timeouts.

Using context.Background() for shutdown. This gives infinite time. Always use a deadline: context.WithTimeout(context.Background(), 30*time.Second).

Not waiting for workers. If you only call srv.Shutdown but your background jobs are still running, they'll be killed mid-operation when the process exits. Wrap workers in a WaitGroup and wait for them.

Same timeout for HTTP and workers. A slow database query in a worker might need more time than an HTTP request handler. Consider separate timeouts or a total timeout that accounts for both phases.

Panicking on shutdown errors in defer. If you call panic() inside a defer during shutdown, recovery becomes harder. Log the error and exit cleanly with os.Exit(1) if needed.

Key takeaways

• Use signal.NotifyContext to catch SIGTERM and SIGINT signals in a single, context-aware call.
• Call srv.Shutdown(ctx) with a deadline—not an infinite timeout—to stop accepting new connections while finishing in-flight requests.
• Pass the same context to background workers so they can check for cancellation in their loops.
• Track workers with sync.WaitGroup and wait for them to finish before exiting.
• Test your shutdown path during deploys; log timings and errors so you can spot problems before they hit production.
• Set realistic timeouts based on your workload; 30 seconds is typical for web APIs, but adjust if you have longer-running operations.