Health checks for Go services: best practices explained

We've all been paged at 3 AM because a service appeared healthy but couldn't actually connect to its database. Health checks for Go services: best practices explained is about distinguishing between a process that's merely running and one that's truly ready to handle traffic. Getting this right saves your team from cascading failures and unnecessary incident post-mortems.

Why health checks matter in production

In a distributed system, a crashed process is obvious—Kubernetes kills the pod and starts a new one. The real problem is a zombie process: it's listening on a port, responding to requests, but it can't talk to Redis or Postgres. Your load balancer sends traffic to it. Requests hang or fail. Users see errors.

Health checks let orchestrators and load balancers make intelligent routing decisions. They answer two critical questions: Is this process alive? Can this process actually do work?

Without proper health checks, you're flying blind. With them, you catch dependency failures before they become customer-facing incidents.

Liveness: the process is still running

A liveness probe tells your orchestrator whether the process should stay alive. If it returns failure, Kubernetes (or your container runtime) will restart the pod. Liveness checks should be minimal: does the process exist? Can I reach its HTTP port?

In practice, a liveness endpoint is almost trivial. It should not check external dependencies. If your liveness check depends on a database being up, you'll trigger unnecessary restarts when that database has a transient hiccup.

Here's a straightforward example:

func handleLiveness(w http.ResponseWriter, r *http.Request) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(http.StatusOK)
	json.NewEncoder(w).Encode(map[string]string{"status": "alive"})
}

http.HandleFunc("/healthz", handleLiveness)
http.ListenAndServe(":8080", nil)

That's it. The endpoint confirms the process is running and can handle HTTP. We don't check the database, cache, or any external service. Liveness is about the process itself, not its dependencies.

Many teams make the mistake of making liveness checks too complex. Keep them fast and stateless. If the check takes more than a second or two, you risk timeouts during high load. If it depends on external calls, you're adding unnecessary coupling and failure modes.

Readiness: dependencies are reachable

Readiness is where we actually verify that the service can do useful work. A readiness check should confirm that all critical dependencies—database, cache, message queue, external APIs—are reachable and responding.

When readiness fails, orchestrators remove the pod from the load balancer but don't restart it. This gives you time to investigate and fix the underlying issue without creating a restart loop.

A real readiness check connects to each dependency and verifies basic functionality:

func handleReadiness(w http.ResponseWriter, r *http.Request) {
	ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
	defer cancel()

	checks := map[string]bool{
		"database": checkDatabase(ctx),
		"redis":    checkRedis(ctx),
		"queue":    checkMessageQueue(ctx),
	}

	allHealthy := true
	for _, healthy := range checks {
		if !healthy {
			allHealthy = false
			break
		}
	}

	w.Header().Set("Content-Type", "application/json")
	if allHealthy {
		w.WriteHeader(http.StatusOK)
	} else {
		w.WriteHeader(http.StatusServiceUnavailable)
	}
	json.NewEncoder(w).Encode(checks)
}

func checkDatabase(ctx context.Context) bool {
	err := db.PingContext(ctx)
	return err == nil
}

func checkRedis(ctx context.Context) bool {
	err := redis.Ping(ctx).Err()
	return err == nil
}

func checkMessageQueue(ctx context.Context) bool {
	// Verify connection to your message broker
	err := queue.Check(ctx)
	return err == nil
}

Notice that we set a timeout on the readiness check. If a dependency hangs, we don't want the entire readiness probe to hang. Two seconds is a reasonable default; adjust based on your network latency and SLA requirements.

The readiness endpoint returns 200 OK when all checks pass and 503 Service Unavailable when any check fails. Load balancers and orchestrators understand these status codes and route traffic accordingly.

Fail readiness during graceful shutdown

When a service receives a termination signal (SIGTERM), it should stop accepting new requests immediately. The cleanest way to do this is to fail readiness checks while allowing in-flight requests to complete.

Here's a pattern we use in production:

type Server struct {
	httpServer *http.Server
	isShutting bool
	mu         sync.Mutex
}

func (s *Server) handleReadiness(w http.ResponseWriter, r *http.Request) {
	s.mu.Lock()
	shutting := s.isShutting
	s.mu.Unlock()

	if shutting {
		w.WriteHeader(http.StatusServiceUnavailable)
		w.Write([]byte("service is shutting down"))
		return
	}

	// ... normal readiness checks
	w.WriteHeader(http.StatusOK)
}

func (s *Server) Shutdown(ctx context.Context) error {
	s.mu.Lock()
	s.isShutting = true
	s.mu.Unlock()

	// Give load balancers time to see the failed readiness check
	// and remove us from rotation (typically 5-15 seconds)
	time.Sleep(10 * time.Second)

	return s.httpServer.Shutdown(ctx)
}

When shutdown begins, we set isShutting to true. The readiness check immediately starts returning 503, which signals to load balancers to stop sending new traffic. Meanwhile, existing requests continue to be processed. After a grace period (10-15 seconds, depending on your load balancer configuration), we close the HTTP server.

This pattern prevents connection errors and partial responses. Clients experience a seamless transition to a healthy replica.

Keep checks fast

Health checks run frequently—sometimes every few seconds. If they're slow, they consume CPU and database connections. If they hang, they can cause timeouts and false positives.

Here are concrete practices:

• Set aggressive timeouts. Use context timeouts for every external call. Don't let a slow database block your health check.
• Cache connection state. Instead of opening a new connection for every check, reuse existing connections and cache their status for a second or two.
• Use lightweight queries. For database checks, run a simple query like SELECT 1 rather than complex transactions.
• Avoid side effects. Never update state, create logs, or trigger metrics in your health check. Keep it read-only and non-invasive.
• Parallelize checks. If you have multiple dependencies, check them concurrently rather than sequentially.

Here's a faster readiness check that parallelizes:

func handleReadiness(w http.ResponseWriter, r *http.Request) {
	ctx, cancel := context.WithTimeout(r.Context(), 2*time.Second)
	defer cancel()

	results := make(map[string]bool)
	var wg sync.WaitGroup

	checks := map[string]func(context.Context) bool{
		"database": checkDatabase,
		"redis":    checkRedis,
		"queue":    checkMessageQueue,
	}

	for name, check := range checks {
		wg.Add(1)
		go func(n string, c func(context.Context) bool) {
			defer wg.Done()
			results[n] = c(ctx)
		}(name, check)
	}

	wg.Wait()

	allHealthy := true
	for _, healthy := range results {
		if !healthy {
			allHealthy = false
			break
		}
	}

	w.Header().Set("Content-Type", "application/json")
	if allHealthy {
		w.WriteHeader(http.StatusOK)
	} else {
		w.WriteHeader(http.StatusServiceUnavailable)
	}
	json.NewEncoder(w).Encode(results)
}

All three checks run concurrently now. The endpoint returns as soon as all goroutines complete or the context times out. This reduces latency significantly.

Common mistakes

• Making liveness checks do too much. Liveness should verify only that the process exists. If you check the database, you'll restart pods unnecessarily when your database has temporary issues.
• Not setting timeouts on health checks. A slow dependency can hang your health check, defeating its purpose. Always use context timeouts.
• Logging every health check. Health checks run frequently and log messages create noise and I/O overhead. Log failures only, not successes.
• Forgetting to fail readiness during shutdown. Without this, traffic continues to route to the pod during termination, causing connection errors. Fail readiness early in the shutdown sequence.
• Using health checks for metrics collection. Health checks should be lightweight diagnostics, not data-gathering operations. Use separate metric endpoints for that.
• Checking too many dependencies. Only check critical path dependencies. If a secondary service is down, should the entire pod be marked unhealthy? Usually not.

Key takeaways

• Liveness checks verify the process is alive; readiness checks verify it can handle traffic. Keep them separate.
• Liveness endpoints should be trivial and dependency-free. Return 200 if the process is running, nothing more.
• Readiness endpoints should check critical dependencies with short timeouts and return 503 if any fail.
• Always set context timeouts on health check calls. Use concurrent checks to keep response time low.
• Fail readiness immediately on shutdown, but allow in-flight requests to complete before closing the server.
• Keep health checks stateless, fast, and non-invasive. Avoid logging every check or using them for metrics.
• Configure your orchestrator (Kubernetes) and load balancers to respect the health check endpoints. Otherwise they're just HTTP endpoints that nobody uses.