Structured logging with slog: Go lessons from production

Opening

We've all been there: digging through logs at 2 AM, grep-ing through plain text strings trying to correlate requests across services, and realizing we have no idea which customer triggered that error. Structured logging with slog: Go lessons from production changes how we approach observability in Go. Since Go 1.21, the standard library gives us slog, a purpose-built structured logging package that replaces the fragmented ecosystem of third-party loggers we've relied on for years.

In production systems—whether you're running e-commerce checkouts, ride-hailing services, or trading platforms—unstructured logs become a bottleneck. By moving to structured logging with key-value pairs instead of formatted strings, we gain machine-readable output that scales with your infrastructure.

Why structured logging matters in production

Log aggregation tools like ELK, DataDog, or Splunk thrive on structured data. When you log "user 42 purchased item 1337", parsing it requires fragile regex patterns that break when your message format changes. When you log structured data with fields like user_id=42 and item_id=1337, any aggregation tool can instantly index and query those fields.

We've learned this the hard way. A few years back, we were debugging a payment failure affecting a small subset of transactions. Our logs contained human-readable messages but no consistent way to extract the merchant ID or payment gateway. We lost hours constructing log queries. Structured logging would have given us instant filtering: {"merchant_id": 1024, "status": "failed"}.

Key-value pairs instead of formatted strings

The core shift with slog is moving from formatted messages to key-value attributes. Instead of building a sentence, you emit discrete fields.

package main

import (
	"log/slog"
	"os"
)

func main() {
	// Old way (still works, but suboptimal)
	// log.Printf("User %d logged in from %s", userID, ipAddr)

	// New way with slog
	logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
	logger.Info("user login",
		"user_id", 42,
		"ip_address", "192.168.1.100",
		"timestamp", "2024-01-22T10:30:00Z",
	)
}

The output is human-readable during development but fully parseable by machines:

time=2024-01-22T10:30:00.000Z level=INFO msg="user login" user_id=42 ip_address=192.168.1.100

Each key-value pair becomes a filterable dimension. In a CRM system, you might log:

logger.Info("contact update",
	"contact_id", contact.ID,
	"operation", "merge_duplicates",
	"records_affected", 3,
	"duration_ms", elapsed,
)

Later, you can ask questions like "show me all merge operations that took longer than 100ms" without touching your log code. The structure is already there.

JSON handler for production

For local development, the text handler is fine. For production, JSON is non-negotiable. JSON is atomic, unambiguous, and every log aggregation system knows how to parse it.

package main

import (
	"log/slog"
	"os"
)

func main() {
	// Development: human-readable text
	// logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

	// Production: structured JSON
	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
	
	logger.Info("payment processed",
		"order_id", "ORD-2024-001",
		"amount_cents", 9999,
		"gateway", "stripe",
		"latency_ms", 145,
	)
}

The output looks like:

{"time":"2024-01-22T10:30:45.123Z","level":"INFO","msg":"payment processed","order_id":"ORD-2024-001","amount_cents":9999,"gateway":"stripe","latency_ms":145}

This single line contains everything your log aggregation pipeline needs. You can ship it to Splunk, DataDog, CloudWatch, or any other service, and it will be instantly indexed and queryable.

In a real e-commerce system we support, switching to JSON logging reduced debugging time by roughly 70% because we could finally correlate order events across the checkout service, payment service, and fulfillment service without string parsing.

Add request IDs with context

In distributed systems, a single user request often touches multiple services. Without request IDs, logs become meaningless spaghetti. slog integrates beautifully with Go's context package, letting you pass a request ID through your entire call stack and attach it to every log line.

package main

import (
	"context"
	"log/slog"
	"os"
	"github.com/google/uuid"
)

type ctxKey string

const requestIDKey ctxKey = "request_id"

func withRequestID(ctx context.Context) context.Context {
	return context.WithValue(ctx, requestIDKey, uuid.New().String())
}

func getRequestID(ctx context.Context) string {
	id, ok := ctx.Value(requestIDKey).(string)
	if !ok {
		return "unknown"
	}
	return id
}

func handleRequest(ctx context.Context, logger *slog.Logger) {
	// Attach request ID to every log in this handler
	logger = logger.With("request_id", getRequestID(ctx))
	
	logger.Info("processing order", "customer_id", 123)
	
	// All subsequent logs include request_id automatically
	fetchUserData(ctx, logger)
	processPayment(ctx, logger)
}

func fetchUserData(ctx context.Context, logger *slog.Logger) {
	logger.Info("fetching user data", "source", "postgres")
}

func processPayment(ctx context.Context, logger *slog.Logger) {
	logger.Info("payment initiated", "gateway", "stripe")
}

Now every log line in that request contains "request_id": "abc123...", even though you only set it once. When a ride-hailing driver's pickup takes an unexpected route, you can grep all logs for that request ID and watch the entire journey from matching to drop-off.

The .With() method creates a new logger with those fields added to every subsequent log. It's not free—it's context-local—but it's the right pattern for per-request data.

Set levels via configuration

Hardcoding log levels is asking for trouble. At 3 AM when your system is melting down, you need the ability to flip logging to DEBUG without redeploying. slog makes this straightforward with handlers.

package main

import (
	"log/slog"
	"os"
)

type Config struct {
	LogLevel string // "debug", "info", "warn", "error"
}

func newLogger(cfg Config) *slog.Logger {
	var level slog.Level
	switch cfg.LogLevel {
	case "debug":
		level = slog.LevelDebug
	case "info":
		level = slog.LevelInfo
	case "warn":
		level = slog.LevelWarn
	case "error":
		level = slog.LevelError
	default:
		level = slog.LevelInfo
	}

	opts := &slog.HandlerOptions{Level: level}
	handler := slog.NewJSONHandler(os.Stdout, opts)
	return slog.New(handler)
}

func main() {
	cfg := Config{LogLevel: "info"}
	logger := newLogger(cfg)

	logger.Debug("this won't appear", "reason", "level is info")
	logger.Info("this will appear", "status", "operational")
	logger.Error("critical event", "code", 500)
}

Load the log level from environment variables, feature flags, or a config file. In production, we typically default to INFO and only bump to DEBUG when investigating specific issues, then roll it back.

This saved us during a SaaS outage: we enabled DEBUG logging on one service, let it run for 15 minutes, then disabled it. Without modifying code or restarting anything. The logs told us exactly where a race condition was hiding.

Common mistakes

Mistake 1: Mixing formatted strings with structured fields. Don't do logger.Info(fmt.Sprintf("User %d logged in", userID)). Put userID in a key-value field instead. The message should be static and human-readable; the data should be in attributes.

Mistake 2: Logging sensitive data. It's easy to accidentally log passwords, API keys, or PII. Even with structured logging, sensitive fields can leak into aggregation systems. Use a custom handler or middleware to redact known sensitive keys. In financial systems, never log full card numbers or SSNs.

Mistake 3: Ignoring cardinality. If you log a field with unbounded unique values (like request IDs or user IDs with high cardinality), your log aggregation system can buckle under query load. Use request IDs (they're designed for this), but be cautious about fields with thousands or millions of unique values.

Mistake 4: Not using context appropriately. Passing logger instances through function parameters instead of deriving them from context makes code brittle. Use context to pass request-scoped data (request ID, user ID, correlation ID) and derive loggers or enriched loggers from it.

Key takeaways

• Structured logging with key-value pairs replaces the fragile regex parsing required by formatted strings.
• Use slog.NewJSONHandler in production to emit JSON that aggregation tools can parse and index immediately.
• Attach request IDs via context and logger.With() to correlate logs across service boundaries.
• Externalize log level configuration so you can adjust verbosity at runtime without redeploying.
• Keep messages static and data in fields; never log sensitive information; be mindful of cardinality.
• slog is part of the standard library since Go 1.21, so there's no reason to reach for third-party loggers in new projects.