Observability in Go: a practical guide for developers

Introduction

You've built a Go API that works perfectly in your local environment, but two weeks after deployment, latency spikes without warning and your team has no idea where to look. Without proper observability, you're flying blind—and that's a position no production team wants to be in. Observability in Go: a practical guide for developers covers three pillars: metrics, logs, and traces. Together, they form a safety net that lets you understand system behavior, diagnose issues quickly, and prove performance to stakeholders.

In this article, we'll walk through the tools and patterns that senior engineers use every day across e-commerce, CRM, and SaaS platforms built with Go. You'll learn how to instrument your APIs with Prometheus metrics, structured logging via slog, and distributed tracing with OpenTelemetry—and how to expose the runtime metrics that reveal the health of your goroutines and garbage collector.

Why observability matters in production Go services

Go's runtime is powerful but opaque without instrumentation. A memory leak might hide for weeks. A goroutine leak can slowly choke your service. A cascading failure across microservices shows up first not in code, but in latency percentiles or error rates. Observability gives you the signal to detect these patterns before users feel pain.

The three pillars work together:

• Metrics answer "what happened over time?"—request rates, error percentages, latency buckets.
• Logs answer "what did the system say when it happened?"—structured data tied to specific events.
• Traces answer "what path did this request take?"—end-to-end visibility across services.

Without all three, you're always missing part of the story.

Prometheus client for metrics

Prometheus is the industry standard for time-series metrics in Go. The official prometheus/client_golang client makes it straightforward to instrument your code. We typically expose metrics on a separate port and scrape them every 15 seconds—this gives you raw data that a Prometheus server then aggregates and stores.

Let's look at a typical API endpoint that tracks request duration, count, and errors:

package main

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"net/http"
	"time"
)

var (
	httpRequestsTotal = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "http_requests_total",
			Help: "Total HTTP requests by method and endpoint.",
		},
		[]string{"method", "endpoint", "status"},
	)

	httpRequestDuration = prometheus.NewHistogramVec(
		prometheus.HistogramOpts{
			Name:    "http_request_duration_seconds",
			Help:    "HTTP request latency in seconds.",
			Buckets: prometheus.DefBuckets,
		},
		[]string{"method", "endpoint"},
	)
)

func init() {
	prometheus.MustRegister(httpRequestsTotal)
	prometheus.MustRegister(httpRequestDuration)
}

func recordMetrics(method, endpoint string, status int, duration float64) {
	httpRequestsTotal.WithLabelValues(method, endpoint, fmt.Sprintf("%d", status)).Inc()
	httpRequestDuration.WithLabelValues(method, endpoint).Observe(duration)
}

func handleOrder(w http.ResponseWriter, r *http.Request) {
	start := time.Now()
	defer func() {
		duration := time.Since(start).Seconds()
		recordMetrics(r.Method, "/order", http.StatusOK, duration)
	}()

	// Handle the request...
	w.WriteHeader(http.StatusOK)
}

func main() {
	http.HandleFunc("/order", handleOrder)
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}

The histogram gives you latency buckets (0.005s, 0.01s, 0.025s, 0.05s, etc.) so you can answer questions like "what percentage of requests are under 100ms?" without storing raw durations. The counter tracks every request—crucial for computing error rates.

In production, we also expose custom business metrics: orders placed per minute, payment processing duration, inventory levels. These metrics often matter more to your stakeholders than raw HTTP stats.

Structured logging with slog

Go 1.21 introduced log/slog, the standard library's structured logging package. Unlike traditional log statements that dump text, structured logs emit key-value pairs—perfect for parsing, filtering, and correlating events across your stack.

Here's a realistic scenario: a ride-hailing backend logs every booking request with structured data:

package main

import (
	"log/slog"
	"os"
	"time"
)

func bookRide(userID, driverID string, fare float64) error {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

	logger.Info("ride_booking_started",
		slog.String("user_id", userID),
		slog.String("driver_id", driverID),
		slog.Float64("fare", fare),
		slog.Time("timestamp", time.Now()),
	)

	// Simulate payment processing
	err := processPayment(userID, fare)
	if err != nil {
		logger.Error("payment_failed",
			slog.String("user_id", userID),
			slog.Float64("fare", fare),
			slog.String("error", err.Error()),
		)
		return err
	}

	logger.Info("ride_booking_complete",
		slog.String("user_id", userID),
		slog.String("driver_id", driverID),
		slog.Float64("fare", fare),
	)
	return nil
}

func processPayment(userID string, fare float64) error {
	// Implementation
	return nil
}

When you emit JSON-structured logs like this, your log aggregator (Datadog, CloudWatch, Loki) can parse the fields instantly. You can filter for all bookings that failed due to payment, group by driver, and spot patterns—all without parsing text.

We recommend using slog in production over third-party libraries. It's built into the language, zero dependency risk, and the ecosystem is catching up with handlers for every popular log sink.

OpenTelemetry for tracing

Traces show the complete journey of a single request through your system. In a microservices setup—frontend calling an order API, which calls payment service, which calls fraud detection—a trace ties all those spans together, revealing where time is spent and where failures occur.

OpenTelemetry is the industry standard. Here's how to instrument a simple request handler:

package main

import (
	"context"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

var tracer trace.Tracer

func init() {
	tracer = otel.Tracer("my-api")
}

func createOrder(ctx context.Context, orderID, userID string) error {
	ctx, span := tracer.Start(ctx, "create_order")
	defer span.End()

	span.SetAttributes(
		attribute.String("order_id", orderID),
		attribute.String("user_id", userID),
	)

	// Call another service
	err := chargePayment(ctx, userID, 99.99)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "payment failed")
		return err
	}

	// Validate inventory
	err = checkInventory(ctx, orderID)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "inventory unavailable")
		return err
	}

	span.SetStatus(codes.Ok, "")
	return nil
}

func chargePayment(ctx context.Context, userID string, amount float64) error {
	ctx, span := tracer.Start(ctx, "charge_payment")
	defer span.End()

	span.SetAttributes(
		attribute.String("user_id", userID),
		attribute.Float64("amount", amount),
	)

	// Simulate payment processing
	return nil
}

func checkInventory(ctx context.Context, orderID string) error {
	ctx, span := tracer.Start(ctx, "check_inventory")
	defer span.End()

	span.SetAttributes(attribute.String("order_id", orderID))
	return nil
}

When you run this with an OpenTelemetry collector (Jaeger, Tempo, Datadog), you get a waterfall view showing that create_order spawned two child spans: charge_payment took 150ms, check_inventory took 80ms. If charge_payment fails, that's immediately visible in the trace UI. You don't need to grep logs or guess—the trace shows you exactly what happened.

The context object threads the trace ID through every function, so all spans automatically correlate. This is why we always pass context.Context through your call stack in Go.

Expose runtime metrics: goroutines and GC

Go's runtime provides deep visibility into goroutine counts and garbage collection behavior. These metrics are often the first signal of serious problems: goroutine leaks, GC pauses that spike latency, or memory pressure.

The Prometheus client includes a built-in collector for runtime metrics. Enable it like this:

package main

import (
	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
	"net/http"
)

func init() {
	// Register the standard Go runtime metrics automatically
	prometheus.MustRegister(prometheus.NewGoCollector())
}

func main() {
	http.Handle("/metrics", promhttp.Handler())
	http.ListenAndServe(":8080", nil)
}

This exposes metrics like:

• go_goroutines — current count. A steady climb signals a goroutine leak.
• go_gc_duration_seconds — GC pause histogram. Spikes here correlate with latency spikes.
• go_memstats_alloc_bytes — current allocated memory. Baseline tells you if you're leaking.
• go_memstats_gc_runs_total — total GC cycles. Combined with pause time, reveals GC pressure.

In one e-commerce incident, we noticed go_goroutines climbing slowly but steadily. Investigation found a goroutine spawned per request but never cleaned up—fixed in two minutes once we had the metric. Without it, we'd have debugged application code for hours.

You can also write custom metrics that depend on runtime data:

var gcPauseMS = prometheus.NewGaugeVec(
	prometheus.GaugeOpts{
		Name: "go_gc_pause_ms",
		Help: "Garbage collection pause time in milliseconds.",
	},
	[]string{"gc_run"},
)

func monitorGC() {
	var m runtime.MemStats
	var lastGC uint32

	ticker := time.NewTicker(10 * time.Second)
	for range ticker.C {
		runtime.ReadMemStats(&m)
		if m.NumGC > lastGC {
			pauseNs := m.PauseNs[(m.NumGC+255)%256]
			gcPauseMS.WithLabelValues(fmt.Sprintf("%d", m.NumGC)).Set(float64(pauseNs) / 1e6)
			lastGC = m.NumGC
		}
	}
}

This gives you finer control and the ability to alert when GC pauses exceed your SLA (e.g., "no request should pause for more than 100ms due to GC").

Common mistakes

• Cardinality explosion: Labels with unbounded values (e.g., user ID, request ID) create millions of time series and crash your metrics system. Use bounded labels only (status code, endpoint, region).
• Ignoring context propagation: If you spawn goroutines without passing context, traces break and distributed logging fails. Always propagate context.Context.
• Fire-and-forget metrics: Recording metrics but never actually querying them. Define dashboards and alerts before you deploy—decide upfront what you need to know.
• Structured logs without strategy: Emitting dozens of keys per log line creates noise. Be intentional: log what you'd actually filter on or group by in production.
• No sampling for traces: Tracing every request at high volume is expensive. Sample intelligently—100% for errors, 1% for successes, 100% for tail latencies above your SLA.

Key takeaways

• Use Prometheus metrics to track request rates, latency percentiles, and error counts—these answer "is the system healthy?"
• Structure your logs with slog to emit key-value pairs that your log aggregator can parse, search, and correlate instantly.
• Instrument with OpenTelemetry traces to see the full request path through your services and identify bottlenecks.
• Always expose Go runtime metrics—goroutines and GC pauses reveal leaks and performance issues before they become disasters.
• Avoid cardinality explosions by limiting label values to bounded sets; design dashboards and alerts before deployment.
• Pass context through your entire call stack to ensure traces, logs, and metrics correlate across your system.