Profiling Go with pprof: real-world lessons and examples

Finding Performance Bottlenecks Before They Hit Production

We've all been there: your API works fine locally, but under real traffic in staging, latency climbs and memory consumption creeps upward. Profiling Go with pprof: real-world lessons and examples is how experienced teams isolate these issues in minutes rather than days. pprof is Go's built-in profiler, and it gives you surgical visibility into CPU cycles and memory allocations without recompiling your service.

Unlike external monitoring tools that sample at 10Hz or show you aggregate metrics, pprof captures detailed call graphs and allocation backtraces. We'll walk through exposing the profiler, running CPU and heap profiles, and visualizing flame graphs—all techniques we've used to cut latency by 40% and fix memory leaks in production-critical services.

Expose pprof on an Internal Port

The first step is making pprof available. Most teams run it on a separate internal port that's never exposed to the internet. Here's how we do it in a typical service:

package main

import (
	"fmt"
	"log"
	"net"
	"net/http"
	_ "net/http/pprof"
)

func main() {
	// Main API server on port 8080
	go func() {
		if err := http.ListenAndServe(":8080", myRouter()); err != nil {
			log.Fatalf("Main server error: %v", err)
		}
	}()

	// Debug server on port 6060, internal only
	debugListener, err := net.Listen("tcp", "127.0.0.1:6060")
	if err != nil {
		log.Fatalf("Debug listener: %v", err)
	}
	log.Println("Debug server running on :6060")
	if err := http.Serve(debugListener, nil); err != nil {
		log.Fatalf("Debug server error: %v", err)
	}
}

The magic line is _ "net/http/pprof". That blank import registers pprof endpoints on the default mux. We run it on 127.0.0.1:6060 so it's only accessible from localhost or via SSH tunneling in production.

In a Kubernetes environment, we'd port-forward to the pod:

kubectl port-forward pod/my-service-xyz 6060:6060

This gives you instant access without exposing the profiler to the network. In staging or production, always authenticate and isolate this endpoint.

CPU Profile: Identifying the Hot Path

CPU profiling shows you which functions consume the most CPU time. Grab a 30-second sample during a load test or real traffic:

go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

This downloads a profile and drops you into an interactive shell. Common commands:

• top10 — Show the 10 most expensive functions by CPU time.
• list functionName — Show line-by-line CPU time for a specific function.
• web — Generate a graph (requires Graphviz).

Let's say we're profiling an e-commerce order service. We run a test that creates 10,000 orders and see this output:

(pprof) top10
Showing nodes accounting for 12.50s, 95.24% of 13.12s total
Showing top 10 nodes out of 45
      flat  cum   %cum     calls  name
     5.20s  7.80s 59.45%    10000  processOrder
     2.10s  2.40s 77.53%    10000  validateCreditCard
     1.80s  1.90s 91.10%   100000  json.Marshal
     0.95s  0.95s 97.33%    50000  database.Exec

So validateCreditCard is burning 2.4 seconds across 10,000 calls—that's 240 microseconds per call. That might be expected, or it might reveal that we're hitting an external service synchronously. Let's dig in:

(pprof) list validateCreditCard
Total: 13.12s
ROUTINE ======================== validateCreditCard in service.go
     2.10s  2.40s (full profile)
    100ms   150ms    45    resp, err := httpClient.Do(req)

Aha. We're making a synchronous HTTP call to a payment gateway inside the order loop. One option is batch those calls, or make them concurrent with a worker pool. That's a real optimization, not premature guessing.

Heap Profile: Finding Allocation Hotspots and Leaks

Heap profiling reveals where your service allocates memory, and over time, where it's leaking. Grab a snapshot:

go tool pprof http://localhost:6060/debug/pprof/heap

This shows in-use memory. For leak hunting, it's useful to compare two snapshots. In production, we might grab one at startup and another after a few hours under load:

go tool pprof http://localhost:6060/debug/pprof/heap -base base.prof

Common heap profile findings include:

• Unbounded slice growth: A slice that keeps appending without cap() is reallocating on every power-of-two boundary.
• Goroutine leaks: Blocked goroutines accumulate if you fire them off without a timeout or cancellation.
• String interning: Parsing JSON repeatedly and keeping strings around unnecessarily.
• Connection pool leaks: Database or HTTP clients not returning connections.

Here's a real example. A CRM system was leaking ~50 MB per hour. The heap profile showed:

(pprof) top -cum
     450.52MB  450.52MB 65.89%     12000  cacheEntry

Digging into list cacheEntry, we found a map that cached request bodies with no eviction policy:

var requestCache = make(map[string][]byte)

func handleRequest(id string, body []byte) {
	requestCache[id] = body  // Never cleared!
}

The fix: add TTL eviction or use a bounded LRU cache. That single change cut memory usage in half.

Flame Graphs: Interactive Visualization

Flame graphs show the full call stack. Each box is a function; width is CPU time (or memory). They're especially useful for understanding cascading calls. Open one with:

go tool pprof -http=:8081 http://localhost:6060/debug/pprof/profile?seconds=30

This opens a browser UI at localhost:8081 with several views:

• Flame Graph: Stacked view showing call hierarchy and time.
• Graph: Directed acyclic graph of function calls.
• Top: Ranked list like the CLI.
• Source: Line-by-line breakdown if available.

In a travel booking system, we once saw that 45% of CPU was spent in a seemingly innocent JSON encoder. The flame graph showed it was being called from a loop inside the merchant API handler. We cached the encoding and cut request latency from 800ms to 120ms.

For production, you can save profiles to disk and compare them later:

go tool pprof -base=/tmp/base.prof -http=:8081 /tmp/current.prof

Common Mistakes

Not capturing enough load: A 5-second profile during idle time is useless. Profile during realistic traffic, or use a load test. Aim for at least 1,000 samples in the profile.

Forgetting to instrument slow calls: If your profiler shows runtime.usleep is hot, it might mean you're I/O-bound, not CPU-bound. Check lock contention and network latency separately with traces.

Comparing profiles with different workloads: A profile taken while handling cache hits isn't comparable to one during cache misses. Keep conditions consistent.

Leaving pprof exposed in production without auth: It's convenient for debugging, but it's also a reconnaissance tool for attackers. Always gate it behind a firewall or VPN, and consider requiring a token.

Profiling in debug mode: Disable inlining and optimizations during profiling can hide real bottlenecks. Always profile in production-like builds (with -O2 optimization).

Key Takeaways

• Import _ "net/http/pprof" and expose it on a private port to make profiling immediately available.
• CPU profiling isolates hot functions and call patterns; use go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30 during realistic load.
• Heap profiling reveals allocation hotspots and leaks; compare profiles over time to track memory growth.
• Flame graphs visualize call stacks and are best viewed with -http for interactive exploration.
• Always profile with realistic workloads and in production-like builds; premature optimization is the enemy of clarity.
• Guard pprof endpoints behind authentication and firewalls in production environments.