OpenTelemetry tracing in Go: what developers should know

Introduction

When your Go services span multiple deployments, a single user request might touch five different handlers across three machines before returning a response—and if something goes wrong, finding where the latency spike or error originated becomes a guessing game. OpenTelemetry tracing in Go: what developers should know is the difference between blind troubleshooting and surgical precision. In this article, we'll walk through configuring tracing from the ground up, instrumenting your HTTP layers, creating custom spans around business logic, and ensuring context propagation so every piece of work ties back to the original request.

Why tracing matters in production

Logs and metrics tell you that something happened. Traces tell you how it happened across your entire system. A request timeout in your API might be caused by a slow database query in a downstream service, or a network hiccup between containers, or lock contention in a cache layer. Without traces, you're looking at dashboards and trying to correlate timestamps. With traces, you see the exact call chain and timings in microsecond detail.

OpenTelemetry (OTel) is a vendor-neutral standard for collecting and exporting trace data. Unlike proprietary APM solutions, it lets you send traces to Jaeger, Datadog, New Relic, or any OTLP-compatible backend. We'll focus on the nuts and bolts: how to wire it up in Go and avoid the common pitfalls our team has seen in production.

Set up a provider—Configure an OTLP exporter at startup

Every trace starts with a TracerProvider. This is the factory that creates tracers, and it needs to know where to send trace data. In most setups, you'll configure an OTLP (OpenTelemetry Protocol) exporter that ships spans to a collector or backend.

Here's a typical initialization function you'd call in your main() or in a startup package:

package telemetry

import (
	"context"
	"fmt"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	"go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func InitTracerProvider(ctx context.Context, serviceName string) (*trace.TracerProvider, error) {
	// Configure the OTLP HTTP exporter
	exporter, err := otlptracehttp.New(ctx,
		otlptracehttp.WithEndpoint("localhost:4318"), // Default OTLP HTTP port
		otlptracehttp.WithTimeout(5*time.Second),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create OTLP exporter: %w", err)
	}

	// Define resource attributes (service name, version, environment)
	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String(serviceName),
			semconv.ServiceVersionKey.String("1.0.0"),
		),
	)
	if err != nil {
		return nil, fmt.Errorf("failed to create resource: %w", err)
	}

	// Create the tracer provider with the exporter
	tp := trace.NewTracerProvider(
		trace.WithBatcher(exporter),
		trace.WithResource(res),
	)

	// Set the global tracer provider
	otel.SetTracerProvider(tp)

	return tp, nil
}

func Shutdown(ctx context.Context, tp *trace.TracerProvider) error {
	return tp.Shutdown(ctx)
}

In your main, call this early and defer cleanup:

func main() {
	ctx := context.Background()
	tp, err := telemetry.InitTracerProvider(ctx, "order-service")
	if err != nil {
		log.Fatalf("telemetry init failed: %v", err)
	}
	defer func() {
		if err := telemetry.Shutdown(ctx, tp); err != nil {
			log.Printf("tracer shutdown error: %v", err)
		}
	}()

	// ... rest of your app
}

The WithBatcher option ensures spans are batched before export, reducing overhead. In production, you'll want to point the exporter endpoint to your actual collector URL (Jaeger, Grafana Tempo, etc.), which is usually configurable via environment variables.

Instrument HTTP—otelhttp wraps handlers and clients

The easiest wins come from instrumenting HTTP at the edges: incoming requests to your server and outgoing calls to other services. The go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp package gives you drop-in wrappers.

For incoming HTTP requests, wrap your handler or router:

package main

import (
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
)

func setupRoutes() http.Handler {
	mux := http.NewServeMux()

	// Your normal handler
	mux.HandleFunc("/orders", handleListOrders)
	mux.HandleFunc("/orders/{id}", handleGetOrder)

	// Wrap the entire mux with otelhttp for automatic span creation
	return otelhttp.NewHandler(mux, "http.request")
}

func handleListOrders(w http.ResponseWriter, r *http.Request) {
	// The context in r.Context() now contains the trace span
	// You can start child spans within this
	w.WriteHeader(http.StatusOK)
	w.Write([]byte(`[...]`))
}

func main() {
	// ... init tracer provider
	handler := setupRoutes()
	http.ListenAndServe(":8080", handler)
}

The wrapper automatically creates a span for every request, captures HTTP method, URL path, status code, and duration. It also extracts trace context from incoming headers, so requests from other instrumented services connect as child spans in the same trace.

For outgoing HTTP calls, wrap your client:

package client

import (
	"context"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func NewOrderServiceClient() *http.Client {
	return &http.Client{
		Transport: otelhttp.NewTransport(http.DefaultTransport),
	}
}

// Usage in your service:
func (s *OrderService) FetchUserPreferences(ctx context.Context, userID string) (map[string]interface{}, error) {
	client := NewOrderServiceClient()
	req, _ := http.NewRequestWithContext(ctx, "GET", "http://user-service/preferences/"+userID, nil)
	resp, err := client.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	// ... parse response
	return prefs, nil
}

Notice we pass ctx to NewRequestWithContext. This is critical—the context carries the trace information, so the outgoing HTTP call is recorded as a child of the current span and any headers needed for propagation are automatically added.

Custom spans—tracer.Start(ctx, name) around important work

HTTP instrumentation is automatic, but you need to trace business logic: database queries, complex calculations, cache operations. Use tracer.Start(ctx, name) to create a span around that work.

package order

import (
	"context"

	"go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("order-service")

type OrderRepository struct {
	db *sql.DB
}

func (r *OrderRepository) GetOrderByID(ctx context.Context, orderID string) (*Order, error) {
	ctx, span := tracer.Start(ctx, "db.query.order")
	defer span.End()

	// Your database query
	row := r.db.QueryRowContext(ctx, "SELECT id, user_id, total FROM orders WHERE id = ?", orderID)
	order := &Order{}
	if err := row.Scan(&order.ID, &order.UserID, &order.Total); err != nil {
		span.RecordError(err)
		return nil, err
	}

	return order, nil
}

func (s *OrderService) CalculateShipping(ctx context.Context, order *Order) (float64, error) {
	ctx, span := tracer.Start(ctx, "shipping.calculate")
	defer span.End()

	// Add attributes to the span for debugging
	span.SetAttributes(
		attribute.String("order.id", order.ID),
		attribute.Float64("order.total", order.Total),
	)

	// Complex logic
	base := order.Total * 0.05
	rush := 0.0
	if order.Priority == "express" {
		rush = 10.0
	}
	return base + rush, nil
}

A few patterns worth noting:

• Always call defer span.End() immediately after tracer.Start so the span duration is recorded even if you panic.
• Call span.RecordError(err) when something fails; this marks the span as an error and helps your analysis tools surface problems.
• Use span.SetAttributes() to attach business context. This helps correlate traces with your business data later.
• Create a package-level tracer once and reuse it. Don't create a new tracer on every call.

Propagate context—Always pass ctx so spans connect

The magic of tracing is that every operation in a request chain ties back to the root span. This works only if you propagate context. In Go, that means passing ctx to every function and using it consistently.

Here's a real scenario: you receive a request, query a database, call another service, cache the result, and return. If you don't pass context through each step, you'll see four separate traces instead of one connected trace tree.

package handler

import (
	"context"
	"encoding/json"
	"net/http"

	"your-module/order"
)

func HandleGetOrder(w http.ResponseWriter, r *http.Request) {
	// r.Context() contains the trace span from otelhttp middleware
	ctx := r.Context()

	orderID := r.PathValue("id")

	// Pass ctx to your repository layer
	repo := &order.OrderRepository{db: dbConnection}
	ord, err := repo.GetOrderByID(ctx, orderID)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	// Pass ctx to your service layer
	svc := &order.OrderService{repo: repo, cache: cacheClient}
	enriched, err := svc.EnrichOrderWithPreferences(ctx, ord)
	if err != nil {
		http.Error(w, err.Error(), http.StatusInternalServerError)
		return
	}

	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(enriched)
}

func (s *OrderService) EnrichOrderWithPreferences(ctx context.Context, ord *Order) (*Order, error) {
	ctx, span := tracer.Start(ctx, "order.enrich")
	defer span.End()

	// Pass ctx to cache
	cached, err := s.cache.Get(ctx, "prefs:"+ord.UserID)
	if err == nil {
		return applyPreferences(ord, cached), nil
	}

	// Pass ctx to outbound HTTP call
	prefs, err := s.fetchPreferencesFromUserService(ctx, ord.UserID)
	if err != nil {
		return nil, err
	}

	// Pass ctx to cache.Set
	_ = s.cache.Set(ctx, "prefs:"+ord.UserID, prefs)

	return applyPreferences(ord, prefs), nil
}

Every database call, cache operation, and RPC should receive the context. Most Go libraries that support tracing (Redis, SQL drivers with OTLP wrappers, gRPC) will automatically create child spans if you use the context-aware methods.

Common mistakes

Creating a tracer per request. Tracers are expensive to initialize. Create them once at startup and reuse them. A common mistake is calling otel.Tracer("name") inside a handler; instead, store it as a package variable.

Not propagating context. If you spawn goroutines or pass requests to queues, pass the context explicitly. Context is not inherited by default in goroutines:

// Wrong: goroutine loses trace context
go processOrder(order)

// Right: pass ctx explicitly
go processOrder(ctx, order)

Forgetting to call span.End(). If you don't end a span, it may never export and you'll have incomplete traces. Always defer it.

Over-instrumenting low-value operations. Creating a span for every tiny operation adds latency and noise. Reserve custom spans for database queries, external API calls, and complex business logic. Let the HTTP instrumentation handle request/response cycles.

Not setting the global TracerProvider. If you configure a provider but forget to call otel.SetTracerProvider(tp), third-party libraries won't see your configuration and will use a no-op tracer.

Key takeaways

• Initialize a TracerProvider with an OTLP exporter at startup and shut it down gracefully in a defer block.
• Wrap HTTP handlers and clients with otelhttp to get automatic span creation and context propagation without code changes.
• Use tracer.Start(ctx, name) to trace business-critical operations like database queries and external calls.
• Always pass ctx through your call chain—it's the glue that connects spans into a single coherent trace.
• Record errors with span.RecordError() and add business attributes with span.SetAttributes() to make traces actionable.
• Keep tracers as package-level variables, never create them per request or operation.
• Test your trace exporter configuration in development before shipping; verify traces appear in your backend.